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ABSTRACT 

This study planned further research aimed at 
understanding how the current version of the Graduate Record 
Examinations (GRE) analytical measure reflects cognitive skills that 
are important to success in graduate education. Several activities 
were completed, including selective literature reviews ott (1) 
cognitive and psychometric research on reasoning; and (2) more 
general educational research on the role of reasoning in higher 
education* The two item types included in the current test 
(analytical reasoning and logical reasoning) were analy ed to 
determine more precisely the kinds of basic skills that underlie 
their solution. Based on these activities a summary was made of the 
major thinking and language skills that seem to be required for 
successful performance on the test. Two major conclusions were 
reached; (1) although there is a considerable body of research on 
reasoning, thinking, and problem solving, there are few 
well-developed models or theories to guide the measurement of 
analytical ability; and (2) although reasoning, critical thinking, 
and other such processes are assumed to be requirements of successful 
academic performance, there is little documentation of the specific 
involvement of these skills in graduate education. Based on these 
general conclusions and other more specific findings, further work 
has been planned. A 10-page bibliography follows the report. A 
simulated solution procass for one analytical reasoning item is 
appended . ( Author /akz ) 
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Abstract 

The purpose of the study reported here was to plan further research 
aimed at understanding how the current version of the GRE analytical 
measure reflects cognitive skills that are important to success in 
graduate education. Several activities were completed, including 
selective literature reviews of (a) cognitive and psychometric research on 
reasoning and (b) more general educational research on the role of reason- 
ing in higher education, The two analytical item types included in the 
current test were analyzed to determine more precisely the kinds of basic 
skills that underlie their solution, Based on these activities a summary 
was made of the major thinking and language skills that seem to be 
required for successful performance on the test* Two major conclusions 
that were reached are that (1) although there is a considerable body of 
research on reasoning, thinking, and problem solving, there are few well- 
developed models or theories to guide the measurement of analytical 
ability and (2) although reasoning, critical thinking, and other such 
processes are assumed to be requirements of successful academic perfor- 
mance, there is little documentation of the specific involvement of these 
skills in graduate education. Based on these general conclusions and 
other more specific findings, further work has been planned and is 
proposed in a separate document, 
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Construct Validity of the ORE Analytical Test.- A Resource Document 

The goal of the study reported here was to understand better how the 
current ORE analytical test reflects basic cognitive skills that ar*i im- 
portant to success in graduate education* This goal was to be achieved by 
(a) reviewing relevant cognitive and psychometric research in order to 
model the basic cognitive processes that underlie the solution of analy- 
tical ability test items and (b) reviewing research on the role of higher 
level thinking and reasoning skills in advanced academic performance. In 
addition to learning more about the current analytical measure from these 
activities, we also hopped to be better able to suggest fruitful directions 
for further research on the construct validity of the analytical measure, 

The report begins (Section I) with a description of the analytical 
test, its purpose, and its current structure* Appendix A presents a 
simulated solution process for one analytical reasoning item, Section II 
contains a brief discussion of construct validity as it applies to the 
test and a sunnsary of Board^sponsored research that relates to the con^ 
struct validity of the measure. Section III lays the framework for two 
selective surveys of research on reasoning, A sunmary is provided of some 
psychological and cognitive science research that bears on the construct 
validity of the test and the identification of skills required by it* 
Section III also contains a discussion of some previous research on the 
role of reasoning in academic performance and also previous research on 
the analytical test* It also summarises some research on the training and 
measurement of thinking skills* Section IV presents an overview of the 
major thinking and language skills likely to be required on the test, 
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given tlio previous edueationali psychological, and cognitive science re- 
search* The final section of tlfee report (V) is a summary of findinp, 
especially as they relate Co pl^snning further research on the analytical 
test. One of the objectives su^ggested for further research is to inves- 
tigate hw the reasoning skills measured by the current analytical test 
relate to the academic demands c=>f graduate school and to the outcomes of 
graduate school training, Anoth^er goal is to identify academic demands 
and outcoiaas that are not repre^sented in the current analytical test, 
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I. The GRE Analytical Test 

The analytical test is intended to assess examinees* developed ability 
to reason with complex information as mighfc be required in any area of 
graduate study, The test does not presume mny specialised academic 
content knowledge on the part of examinees- The current version of the 
test is made up of two parallel 30-minute w actions. Two item types are 
used * analytical reasoning and logical tea_soning. Nineteen analytical 
reasoning and six logical reasoning items ^ccur within each section, 
Within sections, items of each type are arr- -anged roughly in their order of 
difficulty. 

Analytical Reasoning *<Juestions 
Figure 1 displays an analytical rgasoni:3ig item and associated 
questions taken from the GRE 1982-83 Inf orni^ation Bulletin, Questions 
occur in sets of three or more. Each set o^f questions is preceded by a 
description of a situation involving fictitrzLous entities — persons $ places* 
things, or events* Descriptions consist of three to seven statements* 
Descriptions mention relationships and cons^fcraints in relationships among 

the entities referred to in the description The three or more questions 

which accompany a description are independe^^t of each other. That is to 
say, each question poses a problem that has no relation to either 
information given in other questions or to the problems posed by other 
questions, Nonethelessj all questions in a set rely on the same situation 
description* 
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Figure 1 

An Analytical Reasoning Itats ^^nd Associated Questions 



There are seven co-workers, three woman 
men (Juan, Karl, Leonard, and Mark) who 
0, or P. 



(Fran, Gladys, and Helen) and four 
eat lunch only at restaurants N, 



Fran goes to a restaurant only on Wednesdays* 

Gladys and Karl are never able to go to a 
restaurant together. 

Helen will not go to a rastaur«nt unless Fran 
also goes. 

Juan and Mark will not go to t^±ie same restaurant 
together unless Gladys also go^^s, 

Karl will not go to restaurant 0* 



I - If s±k of cha co-workers have lunch together at ona of the three 
restaurants on a Wednesdays who of ^=ha following must be absent? 

(A) Gladys (B) Helen (C) Juan (P>» Karl (E) Leonard 

2. What is the greatest nuniber of the c=o-workars who can go out to lunch 
together on Tuesday at: one of the tl»ree restaurants? 

(A) 3 (B) 4 (C) 5 (D) 6 (E) 7 

3. If Juan and Mark eafi lunch with a gr— oup of the co-workers, which of 
the following must be true? 

I. The group can include no mora than three men* 
IX. The group can Include only two*** women, 
III. The group cannot eat at restaitirant 0* 

(A) I only (B) II only (C) III o^mly (D) I and II only 
(1) I and III only 
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The relationships mentioned In analytical reasoning descriptions in- 
clude i ordering in terms of space , time , or other continuous variables ; 
set membership determined by given eriterlai or cause and effect. Clear 
and simple language is used to describe relationships* All the infor- 
mation required to answer questions is given in a description, though 
examinees must already be familiar with the meanings of common words 
occurring in the description and questions* An attempt is made to avoid 
terminology that is specific to logic as a discipline or to other 
specialized academic areas. Answering questions typically involves estab- 
lishing fewer than six intermediate relationships based on what is already 
stated explicitly* 

The difficulty of analytical reasoning questions is related to the 
number and complexity of different relationships that simultaneously enter 
a solution and, to a lesser degree, to the number of conditions or number 
or entities specified for a single relationship * All analytical reasoning 
questions have a unique answer based on a description* However* for some 
questions there may be alternative schemes for interpreting information so 
as to arrive at the single correct answer. 

Two basic types of questions are developed for analytical reasoning 
problem sets* One type requires examinees to infer new information that 
must necessarily be true given an accompanying situation description* A 
second type requires Inferences about the possibility of a state of 
affairs given a situation description, 

The foregoing information was based largely on the GRE program 
description of items* but there are also some interesting independent 
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observations that might be added. For instance, analytical reasoning 
items tend to have a strong combinatorial reasoning flavor. They are 
pu^zle=like in nature and are deliberately constructed so as not to 
require knowledge of specific situations and events* Training in formal 
mathematics , computer science, and informal logic, or experience with 
other formalisms for representing information are not necessary for sol-* 
ving these items* It could be the case, however s that skills achieved in 
such areas might aid in the solution of this problem type; this is an 
issue that seems amenable to further research, A detailed analysis of a 
solution of the analytical reasoning item in Figure 1 is given in Appendix 
A. 

Logical Reasoning Questions 

These questions test the ability to comprehend and assess logical 
relationships among either statements in a prose passage or to answer 
options referring to a passage. Several questions may be associated with 
a prose passage, but for a given passage questions are independent of each 
other* Figure 2 displays a sample prose passage and two questions based 
on this passage p These materials are drawn from the GRE 1982-83 Infor - 
mation Bulletin (Educational Testing Service, 1982), 

Questions are based on short prose passages (no more than 100 words in 
length) and are drawn from popular publications , political speeches, 
conversations, advertisements, and academic texts in the arts and 
sciences* Understanding passages and questions does not require 
specialized knowledge in any academic field, including logic* However, 
some of the materials and issues that these questions present would not be 
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out of place In textbooks or course materials on rhetoric, critical think- 
ing, or formal logic, 

Logical reasoning questions test the general ability to understand and 
analyze arguments, and more specifically, skills in: 

o recognizing the point of an argument 

o recognizing assumptions on which an argument is based 
o drawing conclusions from given premises 
o evaluating arguments and counterarguments 
o analyzing evidence 

The skills addressed by logical reasoning questions have obvious rele- 
vance to the academic reasoning activities required of graduate students. 
The skills outlined feature the comprehension and manipulation of infor- 
mat ion of a complex conceptual nature. Language comprehension skills * 
particularly reading comprehension skills* are also critical to this item 
type. As will be discussed in more detail later, logical reasoning 
questions require more than basic comprehension skills; they require 
sophisticated mental representation of entailment relationships between 
premises of arguments and conclusions drawn from premises. When we 
consider advanced reading, it becomes difficult, if not impossible, to 
distinguish reading comprehension skills from the kinds of reasoning 
skills required to solve logical reasoning questions, Nonethelesss the 
detailed specifications underlying logical reasoning questions permit a 
well-rounded and workable method for probing examinees* reasoning skills, 
In contrast, GRE reading comprehension questions are not designed to 
systematically tap the same specific set of skills as logical reasoning 
questions. 
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Figure 2 



Two Logical Reasoning Items 



L Mrs* Thomas boarded an airplane and saw another passenger sitting in 
her assigned seat. The passenger showed her his boarding pass, and, 
indeed 5 he was in the right seat, Mrs* Thomas turned to the flight 
attendant and exclaimed, "How do you explain the fact that my assigned 
seat is not available? 11 "Because, Madam s " replied the flight atten= 
dant, "there has been a multiple assignment • 11 

The flight attendant's reply is an example of: 

(A) rephrasing rather than explaining a problem 

(B) confusing company policy with employee practice 
fC) arguing from inconsistent points of view 

(D) failing to provide specific support for a general assumption 

(E) presupposing what is to be proved 

2* State Commissioner of Environmental Affairs: "There is no cause for 
alarm at recent disclosures that cancer^causing agents have been found 
in the air near seven chemical plants in the state, Such substances 
have been found in the air at almost all chemical, petrochemicals and 
pharmaceutical plants in the country* 11 

The Commissioner's conclusion depends on his assumption that: 

(A) experts in the Environmental Affairs department have determined 
that the substances are not harmful 

(B) most of the substances found in the air near the plants were 
organic compounds that are not linked to cancer 

(C) such substances are not causing significant harm at the other 
places where they have been found 

(D) substances used in chemicals petrochemicals and pharmaceutical 
manufacturing are harmless 

(E) similar findings would result if similar tests were made downwind 
of many industrial sites 
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II, Construct Validity and Analytical Ability 

The evolution of the analytical measure has not been driven explicitly 
by a cognitive analysis of the thinking skills that ought to be included 
in the analytical section of the cest , but rather by other more tradi= 
tional psychometric concerns. 

Development 

The early development of analytical item types was concerned primarily 
with establishing the face validity, criterion validity, degree of speed- 
edness, difficulty, and reliability of various item types (Miller & Wild, 
1979), Figure 3 displays descriptions of the various item types that were 
investigated * 

Item types were sought which possessed desirable psychometric 
properties and which exhibited discriminant validity relative to 
performance on the verbal and quantitative sections of the GRE General 
Test* The development plan for the analytical ability section focussed on 
four questions (Miller & Wild, 1979* p. 49): 

1. Will the item types be of appropriate difficulty % reliability, and 
speededness? 

2, Will the item types measure skills that are relatively independent 
of the verbal and quantitative ability measures? 

3- Will the item types have criterion validity? 

4, What combination of item types appears to be best in terms of: 
a) efficiency (high reliability in the available time), b) face 
validity, c) criterion validity, d) independence from the verbal 
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Figure 3 



Item Types Originally Considered for the Analytical Ability Section 
l e X21 (Letter Sets ) 

The Letter Sets item type originated in the French Factor Kit and was 
intended to test inductive reasoning* The item type and the instructions 
were revised to eliminate possible ambiguities, and new questions were 
written* Each question consists of five groups of letters* only one group 
of which is unlike the others in alphabetic arrangement. The examinee T s 
task is to identify that dissimilar group* 

Letter Sets questions seemed potentially useful because they were 
shown to be relatively independent of the verbal and quantitative ability 
measures in earlier studies * Although testing a narrow aspect of reason^ 
ing, Letter Sets were assumed to be efficient , requiring a minimal 
investment of time and students at or above the college level, 

2, X22 (Logical Reasoning and Letter Sets ) 

Since Logical Reasoning questions had previously been pretested in a sin- 
gle module on the GRE populations pretesting of Logical Reasoning alone 
would have been repetitive. However, because one possible combination of 
reasoning item types was foreseen to be Letter Sets and Logical Reasoning, 
the two item types were pretested together to determine their rela- 



Logical Reasoning questions are based on brief arguments or statements 
presenting evidence or opinions. The questions require that the examinee 
recognize unstated presuppositions, logical flaws, methods of persuasion, 
conclusions logically following from given arguments, and the like. From 
earlier pretests, it was known that the item types which had high face 
validity, correlated highly with the verbal ability score. However, in 
combination with an item type such as Letter Sets, it was expected to be 
appropriate as a measure of reasoning yielding a separate score. From 
experiments in the Law School Admission Test program, it was known that 
the item type had high criterion validity, with the highest validity 
coefficient among several experimental item types* 

3* X23 (Analytical Reasoning and Letter Sets ) 

Analytical Reasoning questions had previously been pretested in a single 
module on the GRE population, and thus all necessary information on 
reliability, speededness, and relationship to verbal and quantitative 
ability was available* However, since an appropriate reasoning module 
might include both Letter Sets and Analytical Reasoning, the two item 
types were pretested together to determine their relationship, The Letter 
Sets questions used in X23 were identical to those used in X22. 
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Analytical Reasoning questions are based on brief sets of statements 
expressing relationships among abstract symbols (letters) or sets of rules 
governing processes or procedures having few concrete referents. The 
examinee is asked to draw inferences from and sometimes critically assess 
those sets of statements* 

4* X24 (Evaluation of Evidence ) 

Evaluation of Evidence questions are based on brief narratives estab- 
lishing a situation and conclusion drawn from the facts presented. The 
items consist of bits of evidence that, in relation to the situation 
described, strengthens weaken, confirm, disproves or fail to affect the 
conclusion, Newly developed for the GRE program, Evaluation of Evidence 
provided high face validity, but its statistical characteristics were 
unknown. The present options, which are the same for all questions, made 
the item type likely to be efficient, 

5. X25 (Analysis of Explanations ) 

Analysis of Explanations questions are based on brief narratives estab- 
lishing a situation in which an action is taken in order to have a 
specific effect. A later result, which may or may not be directly related 
to the actions is described in a brief statement, Each question is a 
piece of information that must be evaluated in terms of the facts and the 
result. The five options are the same for all questions* The first 
option that cannot be eliminated is the correct answer. Newly developed 
for the GRE program, the Analysis of Explanations item type provided high 
face validity, but its statistical characteristics were unknown* 

6. Yl (Logical Diagrams ) 

Logical Diagrams, an item type derived from Venn diagrams s presents 
several circle diagrams of possible relationships. Each question consists 
of three nouns, and the examinee is asked to select the circle diagram 
that best characterizes the relationship of the three nouns* Logical 
Diagrams had been pretested in a similar form in the French Factor Kit , 
Having proved to be efficient and somewhat independent of verbal and" 
quantitative ability, the item type had promise as a component of a 
reasoning measure on which a separate score could be reported, It was 
noted * however s that the item type presented problems of multiple 
interpretations and key disagreement. To forestall such problems s the 
development process was expanded to include more than the usual number 
of independent reviews. 

7* ¥10 (Deductive Reasoning ) 

The Deductive Reasoning item type consists of a relatively complex set of 
rules the examinee is asked to apply in solving problems based on dia- 
grams. Deductive Reasoning items had high face validity but unknown 
statistical characteristics. 
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and quantitative maasuraa, and e) appropriateness for both science 
students and humanities and social science students * 
These issues were pursued by research involving pretesting of items, sta- 
tistical analyses of pretests , and surveys of students attempting new item 
types* As a result of this work the Analytical Ability section was 
introduced as part of the GRE examination on an experimental basis in 
October 1977* The item types selected for inclusion included: analysis 
of explanations; analytical reasoning! logical reasoning; and logical dia- 
grams* Subsequent research on the Analytical Ability section is discussed 
later in this report* 

Subsequent Board^Sponsored Studies of the Analytical Measure 
The GRE Board has sponsored a number of investigations that bear 
directly on the construct validity of the analytical measure* A factor 
analysis of the restructured GRE Aptitude Test (Powers & Swinton, IS'81) 
revealed that* as a result of the introduction of three analytical item 
types the new test contained a distinct s identifiable, analytical 
dimension that was highly related to the verbal and quantitative factors 
underlying performance on the test. This analytical factor was defined 
predominantly by the three analytical item types f but not by any of the 
verbal or quantitative types* The results of the study thus provided 
considerable evidence of both the discriminant and convergent aspects of 
the construct validity of the new measure* 

Unfortunately, additional research on the new analytical module re- 
vealed that two of the four item types (55 to 70 items in the analytical 
section) were susceptible to both within-test practice (Swinton, Wild* & 
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Wa lima irk, 1983) and to short-term coaching (Swinton & Powers* 1983) and 
teat familiarization (Powers & Swinton, 1982; 198£) and have since been 
deleted from the test* This state of affairs has, unfortunately, rendered 
much of the early research on the analytical measure not entirely appli- 
cable to the revised measure* which contains only logical reasoning and 
analytical reasoning items, In addition, too few of these two item types 
were included in the original analytical measure to permit a comprehensive 
assessment of their individual validity* The situation was further 
complicated by the fact that these items appeared primarily in the final 
positions in a particular separately timed section of the test, thereby 
confounding item difficulty with rate of work, 

Wilson (1982) summarized the results of a series of predictive valid- 
ity studies in which scores on the original analytical measure were 
included as a predictor, He found that in three of four quantitatively 
oriented fields (all but mathematics) , analytical scores were slightly 
more predictive of first-year graduate grades than were either quanti- 
tative scores or verbal scores. In verbal fields, no consistent pattern 
was noted* In educations the analytical measure was the best of the three 
measures* In history, the analytical and verbal measures performed about 
equally, and in sociology the verbal measure was the best predictor, 

Wilson (1982) concluded that "On balance, these findings suggest that, 
in the fields designated as verbal, the predictive value of the analytical 
score may tend to be about like that of the verbal score, whereas, in the 
fields designated as quantitative, the predictive value of the analytical 
score may parallel that of the quantitative score" (p, iv) , With respect 
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Uo incremental validity, the results were inconclusive, but Wilson (1982) 
suggested that " . . . the analytical score may prove to be somewhat more 
useful as an additional predictor in the quantitative than in the verbal 
areas, . (p, vi) , It should be noted, especially with respect to 
incremental validity s that Wilson's results were based on the original 
analytical test, which contained a large proportion of analysis of 
explanations items, a highly verbally=loaded item type. 

In contrasting examinees' profiles of performance on the three sec- 
tions of the Aptitude (General) Test* Swinton and Powers (1979) found 
strikingly different patterns of abilities among candidates in different 
intended major fields. Predictable patterns were evident with respect to 
verbal and quantitative abilities for students intending to major in the 
humanities s social sciences $ and physical sciences, with the first two 
groups having higher verbal than quantitative abilities and the opposite 
occurring for physical science majors. 

The analytical ability factor (again based on the original four item 
types) did not distinguish particularly across the arts/science continuum, 
but rather provided more discrimination within each general category, 
distinguishing the mora theoretical from the more applied majors. The 
basic, more theoretical majors tended to be higher in analytical ability 
than the more applied majors- For example s within the humanities s philos- 
ophy, religion, and art history were higher than music and art. Within 
the physical sciences, mathematics and physics majors were higher than 
chemistry, earth science* and engineering, 
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In his examination of criterion-related validities fo_- item type part 
scores, Wilson (1984) found consistent differences in the performance 
profiles, as well as the validity, of item types across fields* In the 
primarily verbal fields, the validity was generally higher for the logical 
reasoning item type than for the analytical reasoning type. In the quan^ 
titative fields, the opposite was true, Wilson, however, also found the 
somewhat discouraging evidence that the two currently used analytical item 
types may be more highly related to either other verbal item types or to 
other quantitative item types than to each other, thus casting serious 
doubt as to whether these two analytical item types primarily reflect the 
same analytical dimension, 

Recent Thinking about Construct Validity 
As described above, the development of the analytical test did follow 
a carefully established plan of item type identification and research on 
the face validity and psychometric characteristics of item types * How- 
ever, the development plan that was followed differs in some major ways 
from more recent suggestions on how construct validity might inform test 
development* Messick (1981), for example, points to the central role of a 
behavioral theory and a measurement model in establishing the construct 
validity of a test. Understanding what a test measures rests on some 
explicit or assumed theory of behavior tapped by a test, An adequate 
theory should include not only an account of subskills critical to 
performance on assessed constructs, but also an analysis of how perfor- 
mance on a test is related to performance in "real world" tasks* In 
addition, the measurement model has implications for interpreting the 
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construct validity of a teat. The measurement model not only 
opera tionallzes a construct in terms of item performance and test jcores, 
but also establishes a framework for interpreting the skills of examinees 
with respect to the construct being assessed* 

Frederiksen (1982) outlines seven steps characterizing an ideal con- 
struct validity approach to test development* To quote Frederiksen (1982, 
p. 2): 

...Construct validity implies that the first step in 
test development should be a scientific investigation 
of the performance to be assessed or predicted. The 
results of the investigation should then guide the 
development of both criterion measures and the lesired 
assessment test. The new test should be validated by 
demonstrating that it reflects the same cognitive 
processes as the criterion* 

The seven test development steps suggested by Frederiksen include: 

Step It Develop a theory about criterion performance 

Step 2: Develop a criterion test 

Step 3 i Verify the theory about critericn performance 
Step 4: Develop a test for use in assessment or prediction 
Step 5t Test the construct validity of the assessment test 
Step St Test the construct similarity of the assessment for 

trained (expert) and untrained (novice) groups in the 
skill area under assessment 
Step It Repeat steps 1 to 6 
Frederiksen 1 s steps describe an interactive procedure* starting with 
an attempt to identify the criterion behavior which is the target of 
assessment, followed by (a) the development of an instrument to assess the 
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skills of interest, and (b) an investigation of the connections between 
teat performance and criterion behavior. This approach also considers 
whether test performances mean the same thing for groups of examinees that 
differ in their expertise in the domain under assessment* 

A significant obstacle to implementation of a construct validity 
approach to test development lies in how well educational, psychological, 
and assessment theories concretely inform the test development and test 
use process. To date, growth in the sophistication of cognitive theory 
has had virtually no effect on the development of admissions tests, The 
conceptualisation of aptitude tests for college admissions rests on models 
of cognitive skills that have changed very little in the past 40-50 years. 
While testing programs have shown considerable responsiveness to 
psychometric developments over the years $ the response to advances in 
cognitive psychology has been far. more limited. This is true partly 
because educational and psychological research has only recently evolved 
to the point where sophisticated analysis of thinking and learning could 
be thought to have a significant impact on testing, 
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III. Two Selective Reviews of Research on Reasoning 

Selected Psychological and Cognitive Science Research on Reasoning 
Problem Solving and Critical Thinking 

In many way it is difficult to conceptualize reasoning skills without 
drawing attention to the notion of problem solving more generally. As 
Sternberg (1982) notes, the concepts of reasoning, problem solving, and 
intelligence overlap considerably, and our common notions of intelligence 
include both reasoning and problem solving. In turn, problem solving 
constitutes a more general framework for describing reasoning, since to 
understand and view the operations of the latter we must specify problem 
solving circumstances. The importance of problem solving in the 
conceptualization of higher level reasoning skills is evident in the 
definitions of "critical thinking" which have arisen in the field of 
educational psychology. For example* Dressel and Mayhew (1954) suggested 
that the following abilities are related to critical thinking! 
Defining a problem 

Selecting pertinent information for a solution 
Recognizing stated and unstated assumptions 

Formulating and selecting relevant and promising hypotheses 
Drawing valid conclusions and judging the validity of inferences 
In their information-processing description of problem solving, Newell 
and Simon (1972) call attention to the mental representation of problems 
and to the reasoning used in problem solving* They use the term "problem 
space" to refer to an individual^ mental representation of a problem* 
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"Weil=def ined" problems have certain characteristics which affect the 
reasoning applied during problem solving. A well— defined problem is 
characterised by an initial state of affairs constituting a problem, a 
final state of affairs indicating that a problem has been solved, and a 
set of "operators" or mental manipulations that can be used to create new 



problem states from previous problem states. 



Problem solving is goal-driven, Individuals create mental plans for 



solving a problem, and they search their long-term memory for operators 



and knowledge that will aid in working a problem* Newell (1980, p, 700) 

describes the process of problem solving as follows I 

Given a problem in a problem space, the only way a 
subject can solve the problem is by searching in the spacer 
working out from the current state by applying operators , 
adding new states to the stock to be used at new points of 
searchs evaluating whether the results help, etc. To 
accomplish this search requires performing repeatedly a 
fixed set of functions I 



Search Control : The following functions determine the 
behavior in a problem space while working on a given 
problem. 

Decide to quit the problem. 

Decide if a goal state has been produced » 

Select a state from the stock to be the current state, 

Select an operator to be the current operator* 

Decide to save the new state just produced by an operator, 

These functions operate within a cycle consisting of 
repeating the following three steps t 



1. Select a state; select an operator. 
2* Apply operator to state, producing new state, 
3. Decide if a, goal state is met| decide to quit; 
decide to save the new state. 



Control of search processes during problem solving can involve some 
specific high level reasoning processes which have great generality across 
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problem domains, Newell (1980* p* 703) Indicates the following processes, 

w^hich have been implemented in artificial intelligence computer programs 

Eor problem solving i 

o Generate and Test I Generate in. any way possible (e,g* ? 
systematically or haphazardly; a sequence of candidate 
states, testing each for whether it is the desired state. 

o Heuristic Search % Apply heuristics to reject possible 
operators from the current state and to reject newly 
produced states* remember the states with untried 
operators in some systematic w»y (different schemes yield 
search strategies, such as degch first 8 breadth first s 
progressive deepening s and best first ) . 

o Hill Climbing: Generate and apply operators from the 
current state* select one that produces a state with an 
improved evaluation and move to it* 

o Means-Ends Analysis: Compare Clie current state with 
the desired state to detect difference; use that 

difference to select an operator that reduces or eliminates 
it | otherwise proceed as in heuristic search* 

o Operator Subgoaling * If an Qp&rator cannot be applied to a 
current states set up a subgoaL to find a state in which the 
operator can be applied ; otherwise proceed as in heuristic 
search or means-ends analysis * 

o Planning i Abstract from the prresent state (by processing 
only selected information throughout) and proceed to solve 
the simplified problem; use vhatt ii remembered of the path 
as guide to solving the unabstr-*acted problem, 

THiese problem solving strategies are termeC "weak methods" by Newell 

because their use and utility are not neces sarlly dependent upon a 

des tailed knowledge of a particular problem domain. Indeed * as will be 

d&scussed later in this section s when a gnat deal of information is known 

atoout a problem domain there is often less »aed for high level reasoning 

io problem solving. An awareness of such "-weak methods 11 alerts us to a 
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broad class of reasoning ski! Is that problem solvers might Use to 
integrate and monitor the use of reasoning skills that are more automatic 
The ability to use more specific reasoning skills Is often the control of 
morre general problem solving strategies suc:ri as those which NewaEL,! 
dls cusseSi 

Sternberg's (1982) components of intelligence theory provide a very 
helpful framework within whicH^ to view the purposes and integration of 
different sorts of skills in problem solving^ According to Sternberg 
(1980) : 

A component is an elementary information process that 
operates upon internal representation of objects or 
symbols* What is "elementary" is determined by 
theoretical context : What is called "elementary" in 
one theory at one laurel of analysis might be called 
"complex" in another theory at another level of analy- 
sis* A component may translate a eensory input into a 
conceptual representation > transforrm one conceptual 
representation into another, or translate a conceptual 
representation into a motor output CSternberg, 1977b; 
see also Newell & Simon, 1972), The basic idea is that 
components represent latent abilities of soma kind that 
give rise to individual differences^ in measured 
intelligence and in real-world parf ormance, and to 
individual differences in factor scores as wall, 

Sternberg (1982) mentions five kinds of cognitive components i 

nietac omponent s ; performance components; acquisition components; retention 

components \ and transfer cottposnents. Sternberg's (1982, p* 227) 

description of the various components is particularly apt in terms of its 

discussion of reasoning and pr-oblem solving [processes! 

Me t a c omp o n en t s are hL gher-order control processes that 
are used for planning a course of action, for making 
decisions regarding a "Lternative courses of action 
during reasoning or pr-oblem solving^ and for monitoring 
the course of action, 
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Performance components are processes that are used in 
the execution of a reasanittig or problem— solving 
strategy* 

Acquisition components are processes used in learning 
how to reason or to solve E=3roblems* 



Retention coap orients are pKocesses used in retrieving 
previously s to x/ed knowledges s whether it be knowledge 
needed during reasoning or problem^solving or knowledge 
regarding the reasoning or problem^solving algorithm 
itself. 

Transfer c prop o xi evn ts are processes used in 
generalization * that is? im carrying over knowledge 
from one reasoning or prob^lem-soiving task to another* 

The five kinds of c^omponencs meruit ioned can further be classified a^s 

" general components - » »rre quired for f^^erf prmanee of all tasks within the 

task universe. ». class components , t a i— eqnl-rari for performance of a propeer" 

subset of tasks within the task uni^»^erse and spe_cifjc components . , - 

required for the performance of sin ^g le tasks within the task universe. 1 ™ 

Sternberg's overview ofr components smuggest s the complexity involved in 

conceptualizing reasonjLxig performance e* Clearly Newell ? s "search centres* l ff 

shares important attributes with Ste^rnberg 1 s "nietacomporientSj" and 

Newell f s reasoning processes Involves many of Sternberg's components, flie 

degree of overlap in these two accoir^rits may be more important than thei__ x 

differences. Points of overlap - suggest skills which are important and 

which might be assessed in college a_dmissions tests. For example s 

planning or "executive 1 ' functions mi ght be worthy of assessment though 

they are not explicitly tested in cm Trent verbal* mathematicsj or 

reasoning aptitude tests * 
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A critical factor affecting t=fce relevance of components to problem 
solving Is whether problems are '^well-structured" or "Ill-structured, " 
Newell f s (1980) description of pr— oblem solving is appropriate to 
"well-structured" problems, t,e, # . problems in which all of the information 
required to interpret and to soXv-^e a problem are available in the memory 

of the problem solver or in the i xiitial specification of a problem, Simon 

(1973) has defined "ill-structure = problems as problems in which, 
initially, all of the informations necessary to solve a problem is not 
available to the problem solver. An example of a "well-structured" 
problem Is ati algebraic word prob lem in which the problem solver 
understands botli the statement of the problem and the appropriate 
mathematical formulas to represen t it and, therefore, needs only to 
interpret the problem in terms of an equation, In contrast, an example of 
an "ill-structured" problem is th* m problem of "designing a home," In this 
case there aire many options and p«arameters, which lead dynamically to the 
refinement and solution of the pr^oblenw Problems that are well-structured 
place more emphasis on reasoning gprocesses that resemble deduction, 
Problems that are ill-structured ^^equire not only deductive reasonings but 
also inductive reasoning, i*e,^ tHhe gaining of new knowledge based on 
experience and experiment* In regality, well-structured and ill-structured 
problems are on a continuum, 

A critical factor affecting solution of both types of problems is 
knowledge about a problem and a pi^robiem domain. For an expert, with 
considerable krtotfledge about a prc^blem domain, a problem is more likely to 
be well-structured than for a nov^Ece. Also, the expert is usually more 
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skilled than mm novice problem solver in converting ill-strucdiured problems 

into well-stri^setured ones. An expert 1 s organization of both factual 

knowledge abeu^t a domain and procedural knowledge about prob^Lem solving is 

typically grea« fcer than that of a novice problgn) solver. 

Current c©»gnitive research on eKpert^novice differences points to some 

important issueses that could be considered in designing asses^sments of 

reasoning abil_ity fi The following synopsis of findings drawn from Duran, 

Strieker, and Vard (1982) summarises some of the major e one Icms ions to be 

drawn from res earch on expert— novice differericai in problem ^solving! 

Findi ng 1 : Experts T knowledge in a problem solving 
area is more than knowledge of facts and isolated 
princ iple^. Experts maintain knowledge about how facte 
and p irinciples are organized and related to each othser* 
When * asked to categorize problems according to theii™- 
simll^srity , experts sort problems according to the 
princ iples which different problems exemplify, while* 
novie*^s classify problems as similar on the basis ofrr 
crude similarities related to their statement and 
embodriment (Chi, Feltovich, & Gla s er, 1981), A furfcxrier 
impli»eation of this finding supported by research is« 
that MKparts are more able than novicea to analyze a*,aw 
probl™ems and novel aspects of problems in terms of fczne 
most ^relevant principles (Lesgold s Faltovich, Glaser— s & 
Wang, 1981), 

Findi^mg 2 t Experts are more automatid in their prob lem 
solvi^mg behavior than novices, Given their greater 
knowl^^dge in an area and under standing of how to app ly 
knowi^^dge s experts are more likely to "work forward" in 
probl^sm solving rather than "backward" (Larkin, 1980 3* 
That Sis to say, experts are less likely to start 
workirsig a problem by searching extensively for 
alterr=iative ways in which a final solution might be 
formu31ated unless this is necessary, On the other 
hand » novices are more likely to search extensively -f or 
ways Sin which a problem might be cottciptuialized. 
Hxpert^s recognize quickly and "automatically" what 
^incSEtples apply, and they concentrate from the star^^ 
on waj^T-s in which data fit into known problem solving 
al^rS, trims, As a result of this difference, experts 
ar^j qMjiCker and more efficient In golvlng problems tlSnan 
no , 
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Finding 3 i Experts T development of knowledge in an 
area is an outgrowth of extensive experience which 
leads to development of automatic, but flexible , 
problem recognition strategies and procedural knowl^sdge 
about how to solve problems, Lesgold, Feltovich, 
Glaser, and Wang (1981), for example, concluded th£t=r it 
took a number of years of experience in the field fear 
radiologists to acquire rapid recognition skill in 
their examination and diagnosis of patient x-rays * 
Novice radiologists were more likely to ignore data 
that were important to consider in x-ray diagnosis* 
Lesgold, at alt (1981) also concluded that experts rrsay 
be more flexible in their problem diagnosis strategi_eg 
to account for anomalies encountered in a problem* 

Finding 4 * Novices often maintain naive models of 
knowledge in a problem solving domain* Research by 
Green, McCloskey, and Carammaza (1980), DiSessa 
and by Stsven and Collins (1980) has found that 
students often maintain naive models explaining or 
predicting physical phenomena that affect their 
learning of scientific principles* While naive mode is 
are incorrect, and become displaced by a "correct 11 
scientific model through further learning, the naive 
models nonetheless represent a basic important start ing 
point for acquiring knowledge and problem solving 
ability in a problem domain* 

Finding 5 i Transfer of problem solving knowledge 
across similar problems occurs most successfully onl^yy 
when the relationship between two problems is expllc^^t* 
Simon and Hayes (1976) and Reed, Ernst, and Banerji 
(1974), for example, have found that human subjects ate 
unable to recognise and represent formal similarities^ 
between puzzle problems that are isomorphic to each 
other. Instruction in the formal similarities bet^e«^n 
a new problem and an old problem, however, did lead 
a more efficient and correct solution of a new probl^ern, 

Finding 6 1 Very little research in this area has 
attempted to work with large groups of subjects* Mo^st 
of findings 1 through 5 have been established by 
detailed careful study of problem solving behavior 
individuals or small groups of subjects* 
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Deductive and Inductive Reasoning Research 

In contrast to the richer view of reasoning stemming from a problem 

solving approach, most psychological research on reasoning has focussed on 

a more narrow view of reasoning processes with attention given to specific 

classes of reasoning* For example, in their review of thinking and 

concept attainment j Neimark and Santa (1975) listed the following 

subheadings under the general category of "logic and thought"? 

1 - transitive inference 

2* syllogistic reasoning 

3. comprehension of logical symbols and terms 

4. conditional reasoning 

5. formal operations 

6- generalization and abstraction 

The range of topics mentioned by Neimark and Santa exemplify research 
topics in the area of deductive reasoning, Descriptions of recent 
research are provided in Wason and Johnson-Laird (1972), Revlin and Mayer 
(1978), Johnson-Laird and Wason (1977), Falmagne (1975) , and Anderson 
(1982)* The key questions of research have focussed on whether a human f s 
reasoning behavior conforms with the operation of a well— specif ied model 
of deductive reasoning* Sometimes this latter model has been based 
closely on a formal logical system* while in other cases models have also 
introduced a human decision making component with reasoning described in 
information processing terms, The most obvious stimulus to research has 
been evidence that humans do not typically apply formal rules of inference 
in a logical manner* Researchers have sought to model and understand how 
it is that humans reason with materials drawn from formal logic. Special 
attention, for example, has been given to the solution of categorical 
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syllogisms (e.g. , All A are B. All B are C. Therefore*,,,); conditional 
syllogisms (e.g*. If A implies B and B Implies C s then,,,,); other 
non-syllogistic forms of conditional reasoning; reasoning about 
quantifiers (e.g. * What are different meanings of "All," "Some," or 
"None"); and solution of Venn diagram problems* It is difficult to 
summarize concisely the findings of this research because researchers 
employ different models and conceptions of thu underlying processes they 
study, Anderson (1982, p, 525) nonetheless provides a useful capsule 



We have seen ample evidence for the conclusion that 
humans are in some sense illogical in their deductive 
reasoning. By this, we mean a number of different 
things i 

1* People do not always interpret statements the way 
logicians prescribe they should* For instance, 
they sometimes interpret i_f as if and only if and 
they interpret All As are Bs as A equals B , 

2, They fail to recognize certain logical conclusions 
as valid, either because they have not learned the 
appropriate rule of inference (for instance 
p^q^#q^p) or because they cannot figure out how 
to combine a number of steps in a deduction (for 
instance, in the hard categorical syllogisms). 

3, They accept certain contingent conclusions because 
in part this acceptance may reflect a misinter- 
pretation of the premises (point 1 above) , but in 
part it reflects an inability to find counter- 
examples to the conclusions* 

4, Faced with the difficulties identified in points 2 
and 3, subjects often fall back on various 
heuristics for solving logical problems* These 
heuristics often succeed, but sometimes they do 
not * 



summary i 




To this list we might add that people f s judgment of how conclusions should 
be worded is biased by the manner and order of wording used in the 
premises of syllogistic arguments (see for example the work of 
Johnson-Laird and Steedman, 1978), 

In addition to the phenomena cited above, logic and rhetoric textbooks 
cite a number of other errors in deductive reasonings some of which have 
been investigated sporadically in psychological research. Table 1 lists 
typical deductive reasoning fallacies as given by Biumberg (1976)* 

There are several important and interesting implications of the 
foregoing discussion with respect to our knowledge of deductive reasoning 
and deductive reasoning errors. First, there is a wide range of deductive 
reasoning phenomena relevant to graduate school academic work, and these 
phenomena are not captured by a single test of deductive reasoning, A 
second implication is that many deductive reasoning "errors" are 
systematic | in some sense they don't represent "errors," but rather the 
influence of alternative or irrelevant sources of knowledge and frames of 
reference on reasoning, As will be discussed in the next section, the use 
of a variety of sources of knowledge and frames of reference in reasoning 
is quite natural, and indeed critical to everyday reasoning and to 
inductive reasoning generally, Such forms of mitigated deductive 
reasoning are critical to some of the skills that are most important to 
learning in graduate school and to production of new knowledge in academic 
areas and science. 

While a wide range of deductive reasoning skills may be needed in 
graduate training, given the limitations of assessment situations what can 
be learned about the construct validity of testable deductive reasoning 
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Table 1 

Fallacies in Deductive Reasoning 



Argument ad baculum ("appeal to the stick "). 
This fallacy is usually defined as the 
appeal to force or to the threat of force 
to obtain acceptance of a conclusion. 

Argument as hominem ("to the man" )* This 
fallacy consists in offering statements about 
a person's character or circumstances as prem- 
ises to oppose his conclusion. 

Argument ad ignorantiam ("from ignorance" ) , 
To commit this fallacy is to argue that a con- 
clusion should be accepted because it has not 
been disproved or that it should be rejected 
because it has not been proved. 

Argument ad misericordiam ("appeal to pity" ) - 
Here the fallacy consists in presenting an 
appeal to pity as if it were a proper prem- 
ise from which to infer a conclusion. 

Argument_ad populum ("to the people" ). An argu- 
ment is said to be an instance of the ad populum 
fallacy if it relies on an appeal to the "emotions 
of the crowd" to gain assent to its conclusion. 

Argument ad verecunriiam ("appeal to authority" ) * 
This is the fallacy of appealing to an authority 
in an area that lies outside of his or her com^ 
petence. 

Accident s The fallacy of accident is committed 
when we argue from seme general principle to 
some particular case whose "accidental" 
features make it an exception to the principle* 

Many _g_uest ions » or comply question . This is 
the familiar "have you stopped beating your 
wife" fallacy, It arises when two or more 
questions are asked as one, with the insistence 
that they be given a single yes-or-no answer* 

^ftit_i_o_P^inclpii_("begging the question" ). An 
argtment that has its conclusion as one of its 
premises is said to commit the fallacy of petitio 
princlpii * It begs the question by assuming what 
is to be proved. 



10* Ignoratio elenchl ("arguing beside the 
point" ) * In the most general sense s 
any argument that commits a fallacy of 
irrelevance is guilty of arguing beside; 
the poJnt ? and thus all fallacies of 
irrelevance are cases of ijnora t io , 
In a more specific sense s the fallacy 
is committed when premises are 
addressed to the wrong conclusion , 

11* Equivocation , A fallacy of equivoca- 
tion is committed when an ambiguous 
term is used to shift senses in the 
same argument. 

12. Composition and division . Composition 
and division are paired fallacies and 
represent special cases of equivoca- 
tion. Here the equivocation is on 
general terms, expressions that denote 
wholes or classes. 

13. Amphiboly . This is a fallacy in which 
an ambiguity attaches not to a word but 
to an entire sentence, Such a sentence 
is said to be amphibolous | amphiboly 5 
like ambiguity is relative to context* 

14. Accent , Some sentences vary in roean^ 
ing depending on which of their worda 
are stressed, if a source of the 
ambiguity in meaning lies in stress? 
then the fallacy of amphiboly is some- 
times called a fallacy of accent, 

13* Fallacy of inconsistent premises . 

This is the case when not only are the 
premises not all true, but when in addi^ 
tion they cannot possibly all be true«~ 
in short | when the premises are 
inconsistent or mutually contradictory* 



Blumberg s A, E, Logic; A first course , New York: Alfred A, Knopf* 1976, 
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skills? Here we will discuss steps toward identifying cognitive component 

skills that might underlie general classes of deductive reasoning tasks. 

Our attention for the movement is focused on very elementary information 

processing skills that are required in the application of higher level 

reasoning skills. Carroll (1976) presents the following description of 

syllogistic reasoning and general deductive reasoning in the context of 

factor and information processing models of cognitive ability* 

Factor Rs (Syllogistic Reasoning) involves both 
retrieving of meanings [of words and sentences] and 
algorithms [i,©., rules for solving problems] from 
relevant portions of LTM [long-term memory]* and 
performing in STM [short=term memory] serial operations 
on materials retrieved. Individual differences [in 
performance] could appear in content and temporal 
aspects [e.g*, speed] of both of these types of 
operations. They could also occur in the probability 
that the subject will give adequate attention to 
details of the stimulus material,* (p. 50) 

Factor (General jleasoning ) is very similar to Factor 
Rs (Syllogistic Reasoning) in that it involves both 
retrieval and serial operations* It would be 
distinguished from Factor Rs only with respect to the 
precise types of contents in LTM that are required to 
be retrieved and utilized in the serial operations, In 
the case of Factor Rs, these contents have to do with 
logical characteristics of certain linguistic quanti- 
fiers (all, some , no , etc . ) , whereas in Factor R the 
contents are more general algorithms concerned with 
concrete relations (time* rate, cost, etc*)* and in 
addition the same types of number associations that are 
involved in Factor N (Number Facility), 



^Bracketed material added for clarity, 
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Carroll* s description of F^actor R (General Reasoning) seems particu- 
larly apt for ORE analytical r*— easoning items. The design of a construct 
validity study of the analyCic^al reasoning and logical reasoning study 
could give attention to asges^^lng elementary cognitive skills such as 
those mentioned in Carroll'^ d&Iescription* 

Carroll' s analysis of the & basic information processing skills required 
in deductive reasoning does nO&-t reflect influences on reasoning that would 
be capable of accounting f 0 r fe allacies in reasoning of the sort outlined 
by Blumberg (1976). A di£fe r ^tTTit approach is required to assess these 
latter influences- This appr^a. ach would require looking at higher-level 
reasoning and strategic thlnki^rmg skills that could be represented by 
logical reasoning items. The pearlier discussion of errors in deductive 
reasoning and the ensuing dis£itEussion on inductive reasoning mention some 
of the issues which should iti Q^uture research on reasoning assessment be 
faced * 

Reasoning from Incomplete K np^tj gledge 

Deductive reasonings whia# critical to all problem solving, must be 
supplemented by other forms Qif reasoning in most realistic problem solving 
and learning settings, Probl#ni£ni solving situations that are ill-structured 
do not typically provide enoughrn information at the outset for a well- 
defined problem representation _ The term "induction," which is often 
applied to reaioning that lasujs « to new knowledge, will not be used in this 
section since more explicit ncrt: = ions of this concept will be introduced in 
the context of current cognitiv*^© theory and research, 
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Collins* Warnock, Aiello* and Miller (1975) distinguish whether 
problem solving involves reasoning about "closed" or "open" worlds* A 
closed world is a knowledge domain in which the facts underlying the 
solution of problems are known explicitly by the problem solver* A 
question such as "Are there volcanos on Hawaii?" requires reasoning about 
closed world knowledge for most of us* since we know for a fact that there 
are volcanos on Hawaii* A question such as "Are there volcanos in 
Guatemala?" would be a question requiring open world reasoning for persons 
without enough factual knowledge to answer this question with certainty. 
In this case s a "reasonable" answer, exemplifying open world reasonings 
might be "There probably are volcanos in Guatemala * because I know that 
Mexico has volcanos and because I know that volcanos tend to occur in 
chains along the North American continent*" 

Collins, at al. (1975) describe two generals non^exhaustive classes of 
open world reasoning strategies* Negative inference strategies are used 
to provide "No" answers to questions about hypothetical facts when 
attempts to derive "Yes" answers fail. A "No" answer is given when 
supporting a "Yes" answer is not readily available, Functional inference 
strategies occur when "people. *• figure out what they do not know by 
reasoning from their knowledge about what it depends on" (Collins* at al # , 
1975, p* 400)* An answer to a question such as "Given that it rained 
yesterday, are the lawns wet?" exemplifies this sort of reasoning, 
Collins, et al. (1975) go on to describe negative inferences and 
functional inferences in terms of different sorts of information 
processing strategies | the reader is referred to their paper for further 
details. 
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Collins, at al, (1975) state that the sorts of open world inference 
strategies they observed in students are at the heart of learning to 
reason inductively. In undertaking open world reasoning, students can, 
with feedback from expert tutors, learn to explore a knowledge domain and 
to integrate their developing knowledge of an area* 

In other relevant research* Collins (1977) has investigated the 
reasoning strategies of expert teachers who use the Socratie method. The 
objective of this work was to describe how expert teachers can instruct 
students in learning facts about a new domain of knowledge, By asking 
leading questions, expert teachers instructed students to reason about a 
new domain of facts and to reason about how to integrate accumulating 
facts. Table 2 lists 24 Socratie rules explored by Collins. Rules 1-15 
emphasize students' learning of general principles from known facts, while 
Rules 16-23 teach students 1 to apply general principles, learned 
abstractly to unknown cases, 

Interestingly, exposing students to use of the rules may conceivably 
lead to students acquiring some of the rules as self-motivated reasoning 
strategies, To the extent that such learning is possible s students may 
develop higher-level reasoning strategies that help them to monitor and 
extend the range and depth of their knowledge in a problem solving domain* 
Reasoning skills of the sort outlined by Collins (1977) do not appear to 
be assessed directly at present by any known psychometric instruments - 

There are a number of additional research areas in cognitive science 
that pertain to reasoning from incomplete knowledge. The whole topic area 
is often subsumed under research on schema theory models of cognition — see 
Bobrow and Norman (1975) and Schank and Abelson (1977) for example. 



41 



Table 2 



Rules in Socratic Tutoring 



Rule 1; Aik about a known case , 
If 

(1) it is the start of a dialogue, or 

(2) there is no other strategy to invoke, 
then 

(3) pick a well-known ease and ask what the 
value of the dependent variable is for 
that case, or 

(i) ask the student if he knows a case with a 
particular value of the dependent variable, 
Example ; Ask the student "Do they grow rice 
in China?" or "Do you know any place where rice 
is grown?" 

Reason for Use ; It brings out any well-known 
facts the student knows about such as rice 
growing in China. 

Rule 2; Ask fojMiny factors . 
If 

(1) a student asserts that a case has a particu- 
lar value of the dependent variable, 

then 

(2) ask the student why. 

Example; If a student says they grow rice 
in China, ask why. 

Reason for Use ; This deteriines what causal 
factors or chains the student knows about. 

Rule 3; Ask for intermediate factors . 
If 

(1) the student gives as an explanation a factor 
that is not an Immediate cause in the 
causal chain, 

then 

j (2) ask for the intermediate steps, 
ERIC 



Example I If the student mentions -nonsoons in 
China as a reason for rice growing, ask "Why do 
monsoons make it possible to grow rice in China?" 

Reason for Use : This insures that the student 
understands the steps in the causal chain, for 
example s that rice needs to ba flooded. 

Rule 4 i Ask for prior factors , 

If • 

(1) the student gives as an explanation a factor 
on a causal chain in which there are also 
prior factors, 

then 

(2) ask the student for the prior factors. 
Example ; If the student mentions water as a 

factor in growing rice, ask him "What do you need 
to have enough water?" 
Reason for Use : Same as Rule 3* 

Rule 5; Form a general rule for an insufficient 

factor . 

If 

(1) the student gives as an explanation one or more 
factors that are not sufficient, 

thin 

(2) formulate a general rule asserting that the 
factor given is sufficient and ask the student 
If the rule is true. 

Example; If the student gives water as the reason 
they grow rice in China, ask him "Do you think any 
place with enough water can grow rice?" 

Reason for Use ; This forces the student to pay 
attention to other causal factors. 
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Rule 6; Pick a coimterexample for an 

Insufficient factor , 

If 

(1) the student gives as an explanation one or 
more factors that are not sufficient, or 

(2) agrees to the general rule in Rule 5, 
then 

(3) pick a counterexample that has the right 
value of the factor (s) given, but the wrong 
value of the dependent variable, and 

(4) ask what the value of the dependent variable 
is for that case, or 

(5) ask why the causal dependence does not hold 
for that case. 

Example ' If a student gives water as the 
reason they grow rice in China or agrees that 
any place with enough water can grow rice, pick 
a place like Ireland where there is enough water 
and ask "Do they grow rice in Ireland? 1 ' or "Why 
don't they grow rice in Ireland?" 
Reason for Use ) Same as rule 5, 

Rule 7; Form a general rule for an unnecessary 

factor . 

If 

(1) the student gives is an explanation one or 
more factors that are not necessary, 

then 

(2) formulate a general rule asserting that the 
factor is necessary and ask the student 

if the rule is true, 

Example ; If a student says rainfall is a 
reason for growing rice, ask "Do you think it 
is necessary to have heavy rainfall in order 
to grow rice?" 

Reason for Use ; This forces the student to 
consider the necessity of a particular factor. 



Rule 8; Pi ck a counterex a m ple for an unnecessary 

factor , 

If 

(1) the student gives as an explanation one or 
more factors that are not necessary, or 

(2) the student agrees to the general rule in Rule 7, 
then 

(3) pick a counterixampli with the wrong value of the 
dependent variable, and 

(4) ask the student what the value of the dependent 
variable is for that case, or 

(5) ask why the causal dependence does not hold 
in that case, 

Example ; If the student gives rainfall as a reason 
for growing rice, ask "Do you think they can grow 
rice in Egypt?" or "Why do they grow rice in Egypt 
when they don't have much rainfall?" 

Reason for Use ; Same as Rule 7, 

Rule 9; Pick a case with an extreme value , 
If " 

(1) the student is missing a particular factor, 
then 

(2) pick a case with an extreme value of that 
factor and ask why the dependent variable has 
a particular value in that ease, 

Example; If the student has not mentioned 
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temperature with respect to rice growing, ask 
"Why don't they grow rice in Alaska?" 

Reason for Use ; This forces the student to pay 
attention to any factor he is ignoring. 

Rule 10: Pose a misleading question , 
If 

(1) there is a ease in which a secondary factor 
overrides the primary factors, 

then 

45 
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(2) pose a misleading question to the student, 
based on the fact that the value of the 
dependent variable is different from what 
would be predicted from the primary factors 
above, or 

(3) pose a misleading choice as to the dependent 
variable between two cases in. which considera- 
tion of the primary factors alone leads to the 
wrong prediction, 

Example ' Because the tree cover in the Amazon 
jungle keeps the temperature down to a high of 
about 85 degrees, ask the student "Do you think 
the temperatures in the Amazon jungle reach 
100 degrees?" or "Do you think it gets hotter in 
the Amazon jungle or Texas?" 

Reason for Use ; This forces the student to 
learn about common exceptions, about secondary 
factors i and about the limitations of general 
rules i 

Rule 11; Specify how the variable depends 

on a given factor , 

If 

(1) the student mentions a factor, but does not 
specify hew the dependent variable varies 
with that factor, or 

(2) only partially specifies the relationship, 
thin 

(3) ask him to specify the relationship more 
precisely, or 

(4) suggest a possible relationship to him* 
Example ; Ask the student "Can you say how 

temperature depends on latitude?" or "Dees average 
temperature increase linearly the further south 

■ you go!" 

Reason for Us e; This forces the student to 

|5 specify more precisely the functional relation 
between the factor in question and the dependent 
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Rule 12: Probe for a necessary factor , 
If 

(1) a student makes a wrong prediction of the 
dependent variable because he has not identified 
one or more necessary factors, 

then 

(2) tell him he is wrong, and ask him to formulate 
a hypothesis about another factor that is 
necessary, 

Example; If a student thinks they can grow rice 
in Ireland because of the heavy rainfall, point out 
they cannot grow rice there and ask "Can you make a 
hypothesis about what other factor is necessary for 
rice growing?" 

Reason for Use ! This forces the student to use 
hypothesis formation as a systematic strategy for 
dealing with unexplained problems, ( 

y 

Rule 13; Probe for a sufficie nt factor, 1 
Jf— —————— — 

(1) a student makes a wrong prediction of the 
dependent variable because he treats a factor 
as necessary when it is not, 

then 

(2) tell him he is wrong, and ask him to formulate 
a hypothesis about another factor that might 
be sufficient, 

Example ; If a student thinks they cannot grow rice 
in Egypt because there is little rain, point out they 
can grow rice there and ask "Can you think of what 
other factor makes it possible to grow rice there?" 
Reason for Use ; Same as Rule 12, 

Rule 14 i Probe for differences between two eases , 
— — 

(1) a student cannot think of a factor that could 
account for different values of the dependent 
variable between two cases, 

then 
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(2) ask him to consider what the differences are 
between the two cases that might account for 
the difference in the dependent variable. 
Example ; If a student cannot think of why hey 
can grow rice in China but not in Alaska, ask what 
the differences are between China and Alaska that 
might account for the difference in rice growing. 
Reason for Use ; Same as Rule 12, 

Rule 15; Request a test of the hypothesis about 

a factor , 

If 

(1) the student has formulated a hypothesis about 
how the dependent variable is related to a 
particular factor, 

then 

(2) ask him how it could be tested, 

Example ; Ask the student "If you want to test 
whether distance from the ocean affects temperature, 
would you compare the temperature in January for 
St, Louis with Washington, D.C. or Atlanta?" 

Reason for Use ; By letting the student to test 
hypotheses, it forces him to learn to control 
other factors that might affect the variable. 

Rule 16; Ask for a prediction about an unknown 

case, 

IF 

(1) a student has identified all the primary 
factors that affect the dependent variable, 

then 

(2) pick a case that is either hypothetical or 
unlikely to be known and ask the student to 
predict the likely value of the variable for 
that case, 



Example: If the student has identified the 
factors that affect rice growing, then ask "Do you 
think they can grow rice in Florida?" 

Reason for Use ; This forces the student to use 
the factors he has accumulated in a predictive way, 

Rule 17: Ask what are the relevant factors to 

consider , 

If 

(3) the student cannot make a prediction, 
then 

(2) ask the student what are the relevant factors 
to consider. 

Example ; Ask the student "If you cannot predict 
whether they grow rice in Florida, what factors 
do you need to consider!" 

Reason for Use ; This teaches the student to ask 
the right questions in trying to make reasonable 
predictions about new cases. 

Rule 13: Question a prediction made without 

enough information . 

If 

(1) a student lakes a prediction as to the value of 
the dependent variable on the basis of some 
set of factors, and 

(2) there is another value consistent with that let 
of factors, 

then 

(3) ask the student why not the other value, 
Example ; If the student predicts they grow 

wheat in Nigeria because it is fertile and warm, 
ask him why not rice. 

Reason for Use ; This forces the student not to 
jump to conclusions without enough information. 
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Rule 19; Point out irrelevant factors . 
If 

(1) the student asks about the value of an 
Irrelevant factor in trying to make a 
prediction, 

then 

(2) point out the factor is irrelevant, or 

(3) ask whether the irrelevant factor affects 
the dependent variable. 

Example i If the student asks whether Denver 
or Salt Lake City is further west in trying to 
deeidt which has the colder temperatures then 
point out that longitude does not matter, or ask 
whither longitude affects temperature. 

Reason for Use ; This forces the student to 
learn what is irrelevant , as will as what is 
relevant, in laking any decision. 

Rule 20* Point out an inconsiste nt predicti on. 
If 

(1) a student makes a prediction about the 
dependent variable which is inconsistent 
with any of the values of the factors 
discussed, 

then 

(2) point out the inconsistency, or 

(3) ask whither the value of the factor discussed 
is consistent with his prediction about the 
dependent variable. 

Example : If the student predicts they grow 
rice in Spain after dryness of the climate has 
been discussed, either point out that a dry 
climate is incompatible with rice growing unless 
there is irrigation, or ask how hi thinks they 
can grew rice when the climate is so dry. 



Reason for Use : This reminds the student to 
consider all the relevant factors in making a 
prediction, and insures he understands the relation 
between the factor and the dependent variable, 

Rule 21; Ask for considera tion of a possible value 

if — 

(1) there is a value of the dependent variable that 
has not been considered and which either is 
consistent with several factors or important to 
consider a priori, 

then 

(2) ask the student to consider that value. 
Example : If the student has not considered rice 

as a possible grain in Nigeria, ask him to consider 
it. 

Reason for Use ; This forces the student to 
actively consider alternatives in making any 
prediction, 

Rule 22 i Test for consistency with a given hypothisi 
If 

(1) a particular value of the dependent variable is 
being considered, and 

(2) the values of one or more relevant factors have 
been discussed, but 

(3) whether these values are consistent with the 
particular value of the dependent variable 
has not been discussed, 

then 

(4) pick one or more of the factors that are 
consistent with the dependent variable and ask 
if they are consistent, or 

(5) pick one or more of the factors that are 
inconsistent with the dependent variable and 
ask if they are consistent. 
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Example: If the hot climate and rainfall in 
Java have been discussed, the student can be 
asked "Is the heavy rainfall in Java consistent 
with growing wheat?" or "Are the hot climate and 
heavy rainfall consistent with growing rice?" 

.Reason if or Use - This tests whether the student 
understands the functional relations between the 
various factors and the dependent variable, 

Rule 23: Ask for consideration of relevant 

factors . 

Tf 

(1) a student makes a wrong prediction in a 
particular case, or 

(2) cannot make a prediction, 
thin 

(3) pick the most relevant factor not discussed 
and 

(4) ask the student what the value of that 
factor is for the particular case, 

Example; If the student predicts that the 
average temperature is very hot in Buenos Aires, 
ask if he knows what the latitude of Buenos 
Aires is. 

Reason for Use ; This forces the student to 
consider relevant factors In making a prediction, 
and elicits whether a mistake is due to wrong 
information about a eases W a mistake about how 
the dependent variable varies with different 
factors * 



Rule 24j Trace the consequences of a general rule . 

If ~ ~~ " 

(1) a student agrees to a general rule such as 
Hull 5 or Rule 7, 

then 

(2) ask if he agrees with the consequences of that 
rule in a particular case. 

Example ! If the student states that it is 
necessary to have flat terrain to grow rice, then 
point out that since they do not have flat terrain 
in Japan, they must not grow much rice and so must 
import most of the rice they eat. 

Reason for Use: Same as Rule 5 or Rule 7, 



! 



Collins, A, Processing in acquiring knowledge. In R, Anderson, R, Spiro, & tf. Montague (Eds,), 
Schooling and the acquisition of knowledge , Hillsdale, NJi Lawrence Erltaum, 1977, 342-350, 
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According to Johnson-Laird (1983), reasoning should be studied in 
terras of persons 1 mental models of the situations and topics to be 
reasoned about. The underlying mental models are adapted from relevant 
memory schemata stored in long-term memory. Recently, there has been 
attention to mental models of dynamic syscemsj i»e. s entities and 
situations which systematically change over time (Centner, & Stevens s 
1983)* Also, special attention has been given to the mental models that 
novices develop about domains of knowledge related to academic course 
material likely to be encountered in college, Centner and Steven* s (1983) 
volume, for example, presents several studies of beginning learners* 
mental models of subject matter in physics and mathematics* Special 
attention is given in this work to students* naive mental models of a 
subject area and to misconceptions arising in reasoning* 

Other research investigating reasoning in terms of mental models has 
stressed description of reasoning heuristics which deviate from 
probability theory models or alternative models based on formal 
measurement theory (see, e,g., Tversky and Kahneman, 1983 for a recent 
overview) - All of the work which has been described tends to suggest that 
so called "errors 1 * in reasoning often are not "errors'* per se; they result 
instead from the systematic application of reasoning strategies that are 
deemed appropriate to a mental model for the situation to be reasoned 
about, The underlying mental model of the situation is often richer than 
that conveyed in the statement of a reasoning problem. The model may be 
based on knowledge stored in long-term memory that is explicit to a 
problem solving domain or that is based on a naive interpretation of a 
problem drawn from common sense beliefs. 
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While not exploriiin deptlLn here, there has been a fair amount of 
research to suggest that socioc — ultural background can have a strong impact 
on reasoning correeUy, For example, Luria (1976) and Scribner (1979) 
describe the difficulties faced I by persons with little formal schooling in 
the solution of formal reasonings problems, Scribner (1979) claims that 
ability to solve abstract reader :n problems is possible only if one has 
learned such form of reasoning ^ as a "genre" of thought through the medium 
of formal schooling car some ocfca social medium. 

Psychom etric and In dividual pl ffcdiferences Research on Reasoning from 
Incomplete Knoyle<|| g — ^ _ . 

The material discuieed in tSie previous section has not focussed on 
assessments psychometric theory*^ or differential psychology* The examples 
of research overvigw©din the fcnresent section, however, have focussed on 
assessment and individual dif f et«ences, The work of Sternberg (1977a) and 
Peliegrino and Glagar (1982) exemplify approaches to these issues which 
utilize tightly specified inf ornrmation processing models of task behavior 
in the domain of anaLoglss probliem solving. According to Peliegrino and 
Glaser (1982) expert! gain solvling analogy problems (of the sorts often 
encountered in apt ituA tests) LXs affected by skill in three underlying 
cognitive factors* Tine factor — s include? management of memory load; 
organization of an appropriate d declarative (or conceptual) knowledge base; 
and procedural knowl^dpof task^ constraints. 

Frederiksen and VanN (19781 = ) GRI-sponsored work on the assessment of 
scientific problem solving abillr %y represents a psychometric approach to 
reasoning skills ass egg orient that does not rely on a formal information 
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processing model* A strength of the approach was its emphasis on problems 
with high face validity; it used simulations of problems that might be 
encountered by social research scientists* Figure 5 lists the problem 
types and reasoning tasks presented examinees in the study* 

This work (in part) probed relationships between examinees 1 ability to 
reason in an open-ended fashion as might be required iri real social 
science research situations, and examinaes 1 verbal and quantitative GRE 
scores, and GRE Advanced Psychology test scores* The results of the 
research indicated that several indices of the quality of examinees 1 
open-ended responses correlated slightly better with GRE Verbal scores 
than with GRE Quantitative scores* Scores on the GRE Advanced Psychology 
test tended to correlate lower than GRE verbal or quantitative scores with 
measures of quality of reasoning on the criterion batt^xy of open-ended 
scientific reasoning tests* Three measures of productivity of ideas in 
reasoning on the open-ended scientific tests showed relatively lower 
correlations with GRE test scores, but some evidence off higher correlation 
among each other. The results of the atudy indicated fcliat the reasoning 
performance of examinees on the open-ended test was tapping skills which 
could be measured reliably and which reflected abilities that were only 
partially captured by existing GRE measures, 

In a follow-up study to this research Ward, Frederilcsen s and Carlson 
(1980) compared performance on the original f ree-response version of the 
Formulating Hypothesis test with performance on a raachi-Xie-scorable version 
of the test. In the machine-seorable version examinees were asked to mark 
appropriate responses to problems, where responeaa were^ presented in a 
list format. The same quality and frequency of response measures were 
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Figure 4 

Tests of Scientific Thinking 

1* Formulating Hypotheses (FH ) . Each problem consists of a brief 
description of a research st^^dy, a graph or table showing the 
principal results, and a statemint of the major finding, The 
task is to write hypotheses fchat might explain, or help to 
explain* the finding* The SL=abject is asked to write not only the 
hypothesis he/she thinks Is most likely to be corrects but also 
other hypotheses that should be considered in interpreting the 
data or in planning another tt^rivestigation, 

2. Evaluating Proposals (EP ) ■ E^e subject is asked to suppose that 
he/she is teaching a senior course in design and methodology 5 and 
that as a class exercise has asked the students to write brief 
proposals of research studie^s * Several proposals presumed to 
have been written by the stucMents are presented as the test 
items* The subject's task i& to write critical comments for each 
student regarding the design,* methodology, or theoretical 
position of the proposal, 

3* Solving Methodological Problfems (SMT ) , Each problem is a brief 

statement of a methodological problem encountered by a graduate 

student or psychologist in pi inning a research study. The 

subject f s task is to write gLt^ggested solutions to the methodo- 
logical problem, 

4. Measuring Constructs (MCH ) , Each problem consists of a name and 
definition of a psychologicaL construct (e,g,, conservatism, 
bigotry, leadership). The ta sk is to suggest methods for 
eliciting the behavior so tha. * it can be observed and measured 
without resorting to ratings or self-report methods. 



Frsderiksen, N. , & Wards W, C. Measuri 
scientific problem-solving. Applied 
2(1), 1-24. 



is for the study of creativity in 
Psychological Measurement , 1978, 
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u^cilized in evaluating performance on the two teats* Irlxi addition to the 
F«=3rmulating Hypothesis test, subjects were administered I a battery of tests 
representing eight cognitive factors. These factors LHa_eluded: 
ve^cabulary, quantitative reasonings induction, logical r xeasoning, 
e<==>gnitive flexibility, expressional fluency, ideational fluency, and 
kr-r^owledge of psychology* 

The results of the study suggested that the frae-r^&sponse and 
Tn^achine-scorable versions of the Formulating Hypothesis test were not 
alLternate forms of the same test. Quality of response fttneasures on the 
m^^ehine-seorable version of the teat appeared to constitttute information 
ab^out examinees 1 aptitudes that was already measured hy the GRE verbal and 
qittantitative measures and by the GRE Advanced Psychology^ test. Quality of 
fe^ sponse measures was also found to be related to fact^tt measures of 
kn^owledge of psychology, induction, and logical reasari£tWTig s Quality of 
re sponse measures on the free-response version of the f formulating 
tfy-"po thesis test contributed little information not aXr#a^ady available on 
th* e maehine-scorable version of the same test* 

The various frequency of response measures, on the £ r ^ee-response 
ve^rsion of the Formulating Hypothesis test, however, did i not show a very 
hi^gh relationship to measures of the same kind in the itfa.^chine-scorabie 
venrsion of the test. Frequency of response measures or* r the f ree=response 
te^^t showed relatively higher correlations with factors r representing 
expressional fluency and ideational fluency. The GRE sCo* ores did not 
associate as strongly with the expressional fluency and t ideational fluency 
fac=tors. The outcome of the research thus suggests that reasoning from 
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incomplete knowledge in Formulating Hypothesis items draws on some 
reasoning skills that are not currently represented by GRE measures and 
that these skills involved the ability to produce numerous ideas about the 
nature of the problem situations represented by items. However, the 
utility of such ideational fluency skills for graduate school performance 
has not been demonstrated* 

Frederiksen, et al. (1981) investigated performance of first year and 
fourth year medical students on a set of branching formulating hypothesis 
problems, Some problems were presented in a multiple choice structured 
response format, Some problems had a medical problem solving content, 
whereas others did not* Subjects were also administered a battery of 
cognitive ability tests * and performance on this battery led to 
identification of six factors i reasoning, verbal comprehensions medical 
knowledge* ideational fluency, cognitive flexibility* and science 
achievement. The results of the study suggested that fourth year medical 
students rely noticeably on knowledge of medicine in both the free- 
response and structured tests as would be expected* On the f ree-response 
version of the medical problem solving test, students rely more on 
reasoning and verbal comprehension than they do on cognitive flexibility 
and ideational fluency, 

Both first year and fourth year medical students showed an association 
between their ideational fluency factor scores and scores of their ability 
to generate numerous ideas on the free-response version of the medical 
problem solving test* In addition, fourth year students showed an 
association between their reasoning factor scores and scores of their 
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ability to generate numerous ideas on the f ree-response teat. The results 
support the hypothesis that as medical students gain in their knowledge of 
medicine they are more likely to rely on reasoning strategies based on 
their knowledge of medicine and that reasoning skills are also important 
to non-medical problem solving that involves generation of a hypothesis. 
Regardless of number of years of medical study, subjects showed an 
association between their propensity to generate a hypothesis in non- 
medical problem solving and their ideational fluency or cognitive 
flexibility. 

Ward, Carlson, and Woisetsehlager (1983) investigated the feasibility 
of a new "ill-structured" logical reasoning item type for the GRE 
Analytical Section test. The new ill-structured items were developed from 
earlier research on maehine-scorable versions of the tests of scientific 
thinking (Ward, Frederiksen, & Carlson, 1980). The new item type 
presented examinees with a several sentence description of an everyday 
situation accompanied by a graph or table showing numerical relationships 
among variables described in the situation description. In addition, a 
finding and a question concerning a conclusion about the finding were 
stated. Examinees were required to select the best multiple choice answer 
option for the question. The new items did not require any specialized 
knowledge of an academic domain for their solution ; although the items 
were multiple choice in nature s examinees needed to generate a number of 
intermediate inferences in order to arrive at an optimal conclusion to 
answer a question. In this regard these items resembled Simon's (1978) 
notion of ill-structured problems. 
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Wards et al. (1983) found that performance on ten of the new items 
correlated . 73 with performance on ten GRE logical reasoning items that 
had been classified beforehand as "ill-structured. " Performance on the 
new items correlated .68 with performance on ten other GRE logical 
reasoning items that were judged beforehand to be "well-structured- " 
Performance on all three of the aforementioned items correlated slightly 
lower with scores on the letter set* 8 inductive reasoning test. 

Ward, et al- (1983) concluded that performance on their new item type 
did not appear to tap skills that were different from those already 
assessed by the existing logical reasoning item type which showed 
"well-structured" problem solving. They did note that their new item type 
might be used to broaden the pool of item types for the present analytical 
section, without altering the constructs under assessment. Ward s et al, 
(1983) also pointed out that their results did not imply that there are 
not other ill-structured item types which would Indeed assess skills not 
represented by items on the analytical section, They also indicated that 
evaluation of connections among steps that examinees take to solve 
ill-structured problems could lead to other useful measures of reasoning 
performances not represented by the current GRE analytical measure. 



61 



-48- 
Th e Role of Thinking in High_er Education 
Thinking as Both an Outcome and a Prerequisjt e 

Intuitively s many of the tasks that arm t squired of both undergraduate 
and graduate students involve some CQmponerie *• of reasoning that may be dis- 
tinct from both general verbal and general gu^antitative skills* In both 
the social and physical sciences, knowledge jf the logic underlying 
experimental designs and research paradigms arad such central concepts as 
causality and correlation would seam important^. The manipulation of 
independent variables, the arranging of exper^^mental conditions, and tha 
drawing of appropriate conclusions from the resulting circumstances are 
central to scientific research. In mathematics s s deductive reasoning is a 
key ability in proving theorems, understanding^ set theory, etc, and in 
learning foreign languages * rule learning may be involved. Many 
disciplines entail the development and evaluate ion of theories, which are 
involved networks of relations between conceft^s, many of which are 
abstract in nature. Examples of such theorizes can be found in numerous 
fields—kinetic theory, Gestalt psychology, tfaa theory of evolution, and 
Keynesian economics (Ennis, .1962), Ennis CI96S2) has laid much of the 
foundation in education research for thinkiiig about critical thinking, 
which, defined as "the correct assessing of -atements" (Ennis, 1962), is 
at the heart of much scholarly work, Ennis (l 962), in a most thoughtful 
analysis, listed twelve aspects of critical t?h _ inking as follows: 
1 - Grasping the meaning of a statement. 

2. Judging whether there is ambiguity in a line of reasoning, 

3, Judging whether certain statements coioitradict each other. 
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4, Judging whether a conclusion follows necessarily. 

5, Judging whether a statement is specific enough* 

6, Judging whether a statement is actually the application of a 
certain principle, 

7, Judging whether an observation statement is reliable, 

8, Judging whether an inductive conclusion is warranted* 
9* Judging whether the problem has been identified, 

10* Judging whether something is an assumption. 
IK Judging whether a definition is adequate, 

12, Judging whether a statement made by an alleged authority is 
acceptable . 

Reasoning skills are recognised both as prerequisites for and intended 
outcomes of post-secondary education* Bowen (1977) for one has euggeit«d 
that among the ten most desirable outcomes of higher educattoii ie the 
broad goal of "rationality , " which is characterised by: 

o the capacity for logical thinking based on useful assumptions 

o the ability to see facts and events in an objective manner, distin- 
guishing the normative f ideological, and emotional froffi the 
positive and factual 

o the disposition to weigh evidence, evaluate facts and ideas 
critically, and to think independently 

o the ability to analyse and synthesize, 

Hartnett and Willingham (1979) also point out educators 1 lMj-standtTig 

contention that one of the most important outcomes of higher eitication %,m 

a developed ability "to think critically and analytically. 11 

"Beyond the acquisition of knowledge, it has often been 
claimed that what one should hope to garner from 
education, is an ability to reason carefully, to recogni%€ 
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valid and invalid inf erances and assumptions? €o approach 
problems with an attitude ^f inquiry that defli^giji evidence 
in support of assertions s to identify logical contradic- 
tions in arguments, and so on," 

Thinking skills are also recognized as importMC JnstructiqM»l 
outcomes long before students reach the advanced itwila of high#t— 
education* The College Board (1983) has undertaken a ten-year #f1==fort to 
strengthen the academic quality of secondary education* With m 
as sis tone e of high school and college faculty s the Board has d^fi^ned 
several basic academic competencies —broad, genetally applicable 
intellectual skills thought to be essential for effective partial JLpation in 
all fields of college study and without which "knowledge of histfo ory ; 
science, language, and all other subjects is unattainable 11 (p* D • These 
basic competencies are reading, writing, speaking a^d listening f 
matheinatice , studying, and reasoning. 

Under reasonings the following abilities have been Identified It 

o The ability to identify and formulate problfiini, as well 
as the ability to propose and evaluate wayg fco solve 
them * 

o The ability to recognise and use inductive at^d deductive 
reasoning g and to recognise fallacies in reasoning. 

o The ability to draw reasonable conclusions fr*o! 

information found in various sources, whether bitten % 
spoken s or displayed in tables and graphs, attd to defend 
oue f s conclusions rationally. 

o The ability to comprehend, develops and use concepts and 
generalizations * 

o The ability to distinguish between fact and § pinion . 

Arons C1979) identifies 16 reasoning capacities greeted of £Qe>llege 

students; these capacities are listed in Figure C.l, Aren's lis* is 




intended to be suggestive of the kinds of skil Is which entering college 
students are expected to utilize. The list is rich in its citation of 
examples of various kinds of reasoning required in dealing with academic 
subject matter- 
Other examples of explicit objectives related to thinking and 
reasoning skills can be found in various collections of instructional 
objectives. As one example, the Instructional Objectives Exchange (1971) 
has presented objectives dealing with "one sid« of judgment: deductive 
logic and assumption recognition, 11 They are: 

o Given a series of statements (either emotionally laden or not) 
which are expressed in various conditional logic formats * the 
student will discriminate which of the conclusions are logically 
valid or invalid 

o Given statements expressed in clags_ logjL o formats (either 

emotionally laden or not) the student wr.ll discriminate valid and 
invalid statements 

o Given a set of statements 5 the student will recognise whether or 
not particular assumptions are necessary and will be able to 
identify assumptions underlying argument m 

o Given descriptions of observations, the student will choose the 
observation with the highest reliability- 

On yet another front, in a strongly worded statement the Commission on 
Higher Education Issues concluded that no studetTit who has not demonstrated 
fundamental competencies in five areas, IncludLmg reasoning, should be 
admitted to full participation in any baccalaur- eate degree. 
Role of Thinking in Reading and Writing 

An ability to comprehend verbal material and a facility for written 
expression are two very general skills commonly expected of virtually all 
students. Exactly how these two general verbal skills relate to reasoning 
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Reasoning Skills Expected of College Studinti 
fecopitlonj Identification) and Control of Variables Ability to Paraphaae FariiriPbi&f Mt in One's Qm ifcia'de 



It is generally expected that students can recognize and eontrol s 
or weigh variables in setting up experiments or in solving verbal 
problems In the natural and social sciences. For example, they shotiid 
be able to account for the possible influence of length, diameter, and 
composition in the well-known Piagetian task of the bending of rods^ 
then control two of these variables in establishing the effect of the 
third, Similarly, the interpretation of historical phenomena requires 
recognition and sorting of political and social factors within the 
information availablij following which the student must decide whither 
cauae B and*effect relationships are discernible or whether the variables 
are so confounded as to preclude a reliable inference, Economics, 
political science, and experimental psychology all depend upon student 
facility in dealing with similar considerations of variables and their 
relations* 

Arittaatieal Reasoning 

Ratio reasoning is required for predicting the alteration of 
gravitational or electrical forces between point objects with changes 
of masSj charge, or spacing, Similarly, the use of scale ratios is 
necessary for interpreting maps, for determining sins of objects 
viewed under a microscopy or for comparing relative changes in vol" 
unes and surface (or cross-sectional) areas with changes of scale, 
Further, it is useful to relate such arithmetical changes to, say, the 
disproportionately large cross-sections of bonis in larger animals, or 
to the rapidity of the dissolving of a material under finer subdivi- 
sion* lie use of similar kinds of ratio reasoning in connection with 
demographic data may be required in political science or sociology, 
and in connection with scaling of factors or rates in economic problems 

Interpretation of division is needed in dealing with concepts of 
densltyj velocity, or acceleration in physics; moles, gas behavior in 
chemistry; and in the study of population and other growth rates in 
, Combinatorial problems arisi in elementary genetics. In prob* 
considerations } in control of variables, and in design of 
experiments. 

Stateaents 



Foraation of intelligible propositional statements requires an 
intuitive grasp of the rules of logic, and of the grammar in which 
such statements are to be madi* For example) taing or understanding 
verbal statements involves Inclusion, exclusion, and serial ordering* 
In addition, one must grasp syntactical constructions such is double 
negatives, subjunctive mood, and the capacity to deal with elemen 5 
tary one* or two-step syllogistic reasoning, lis is not meant to 
include involved propositional logic in which one is forced towards 
symbol! or Boolean algibri for elucidation. The telle skill indi- 
cated, however ? applies to all areas of study which require the use of 
language. 



This expectation is applied to all areas of aM^, A word of 
warning Is needed here; stufemj be able to rephrase a paragraph 
using language similar to tWt In the text without iifidesrstanding its 
content, Thus the insistence ti they put It in their own words is 
of critical importance. 

Awareness of Gaps In Rnotfliigt er Infowtian 

This problem has two dpyii-gaps in the stutafit f i own knowl- 
edge, or incompleteness of Information in a given area of study. 
In the former case, it is erflrti that when a student Hails to recog* 
nize the meaning of a word pr pbol used In an oral presentation or a 
passage of a text* he or sha Vill sense the need for establishing its 
meaning, and have the motive to do whatever is necessary to estab- 
lish it, 

When the problem is incpm^leiss of information ten a particular 
context, the student should T uh that a definite conclusion cannot 
be reachedj or should note tht conclusions or decisions are being 
reached in the face of IncoiffUMrta and hence that iiach conclusions 
must be qualified accordin@iy* Ms overlaps with protfilims of psycho* 
logical maturity, upon whieh tepfe the capacity to «eognize and .1 
tolerate ambiguity in the nateri! under study. j; 

i 

In a sequence of development of a given subject matter, the student 
is expected to gradually dts ttxiguish what has become kncowi or clearly 
established at any particular point from whit has not teen so estab- 
lished. This Implies learning to anticipate some of tine questions still 
to be asked, 



MefsEanding the Kted 



Definitions 



In general, students are feipted to learn the criteria by which it 
is possible to determine whether or not a definition is operational! 
These criteria include; riailiiphen a concept In t passage of text 
has not been clearly defined? recognizing the necessity of using only 
words of prior definition in fonhg a new definition; fe»icoming aware 
of the appeal to shared expgftfiin forming operatiotol definitions. 

ftanalating Harts into Urittft |iMs and trittan SfifamlB 
Into Words 

The skills necessary for sutl operations are more fL,__gorous than 
those needed for paraphrasinf dial passages* Sample: ,s include! 
traniforming a verbal stategenc Ills its equivalent aritTJimetical, 
algebraic, or graphical fori in ay of the natural or 10-cial sciences; 
interpreting a graphical presentation or the results of « a symbolic 
problem solution in words , e^rtiog its content vliHa expressing the 
relevant qualifications and restrictions* 



mi 
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Mseriiloittag tebtm ftsirotian and Inftnnea 

Student! in ill aeadeolc disciplinta must learn to recogniii 
the observational, empirical, or experimental facts that an avillabli 
In a mt presentation or in laboratory work* The next itip la to 
separati thin clearly from the concepts that Hay bt formed or the 
Inferences that my be drawn from them, An exanipla of thii would be 
identifying obsirved facts concerning the extent of illumination of 
the aoon relative to Its position with respect to the sun and then 
separating thesi from the inference that moonlight is reflected 
eunlighc* Another example would be distlngulehing the observed be- 
havior of electrical circuits (that bulbs git diniir as more are put 
in aeries at the same source) from Ehi concept of "resistant" which 
is induced from the observed behavior. In this particular else, 
further distinctions would then need to be nidi between inferences 
concerning the nature of electrical resistance, and the predictions 
that can be made concerning phenomena in more complicated circuits, 

Other examples from quite different areas of inquiry include; 
separating Menders observations of nearly Integral 'ratios of pop- 
ulation members having different color or size characteristics from 
the inference of discrete elements controlling inheritance; the 
dls tine t lon s B conmon in the study of llteraturi^bitween analysis of 
the itracfure of a novel or poem and an interpretation of the work; 
and the historian's task of recognizing the distinction between 
primary historical data and his own interpretation of such date, 

Analytic a Line of taming to Tins ef toderlying AsOTptiosi 

Every line of reasoning has an underlying set of assumptions 
iiparite from the factual data it may utilize* Students need to 
develop the capacity first to discover and second to distinguish 
among assumptions, assertions, and reasoned conclusions, 

Sfiwifig Infirences frn Data and Evidence, Including 

Separate from ths analysis of another's line of reasoning comes 
the formulation of one 1 ! own. For example, given the observation that 
the spot formed on the screen by the cathode beam (Thompson 1 s tjcparl^ 
Hint) remains coherent (Lt n does not smear out) under both electric 
and magnetic deflection, whit inferences can be drawn concerning the 
origin and character of the beam? Or; given the code of law? k 
Hammurabi, whit"if anything-can be inferred about how tht peopk 
subjtct to it lived, and what they held to be of value? Yet another 
iKample is the problem of recognising possible functional or causrand^ 
effect relationships in either positivi or negative eorrelitions in 
the face of statistical scatter or uncertainty; for example, discern* 
ing relative or competing effects of light, heat s or moisture on a 
biological population (with a simultaneous awarensss of whether or 
not the variables hive been adequitely controlled), 

AMllEy to Biseriilnate tetweo Inductive and Betativi 



Studints should be able to follow inductive reasoning used in 
generating i concept, model, or theory^ and use deductive reasoning 
la testing tht validity of a construct, Ihey should perceive the 



analogous patterns of leientlfic thought arising in such broadly 
diverse areas as the Newtonian Synthesis, wave versus particle models 
of Ught, atomic-molecular theory, gene theory, theory of evolution, 
economic or sociological models of society or its parts, and so en, 



Students should be able to visualize, 



In the abstract! outcomes 



that might stem from changes imposed on a given system or situation, 
whether it be In scientific, literary, historical, economic, or 
political contexts, and effect such visualizations through reasoning 
within the basic principles or other rational constraints applicable 
to the system, 

PerfofiiB| QKlitatifij HieoaiDglogical Reasoning or Slinking 

In science and mathematics, the ability to recall formulas and 
manipuliti them algebraically does not by Itself Indicate complete 
understanding of a subject area, Students should also be able to give 
qualitative explanations of principles and make direct inferences 
from them without referring to the" results of numerical calculations, 
Ihey should be able to apply phenomenological reasoning without relying 
on mathematical formalism, 

Checking Ssftrencis, Conelmions, or fesulte 

Skills In this category include i testing for internal consistency; 
using alternative paths of reasoning; examining extrimt, limiting, or ' 
special cases, ' 

In some instances , only Initial or preliminary levels of the skills 
listed in the preceding ssctlon are actually presupposed in college 
work at introductory levels, while enhancement and further development 
of such skills are often implicit objectives of the courses of instruct 
tion, In addition to these objectives! others ire at least implied, 
when not explicitly articulated, in most statements of the cognitive 
pals of higher education, too of these more general goals which 
subsume many of the preceding objectives can be articulated as follows, 



ng SeiMonscloumiis Concinsing Oni f § Ora Blinking 
mi Reasoning hoceiiis 

It is generally desired that students learn to become explicitly 
awari of the modi of reasoning being used in particular situations, 
Ibis provides the basis far consciously seeking to transfer familiar 
modes of reasoning to new and unfamiliar situations, In general, 
studints should Ham to attack a new problem or unfamiliar situation 
by first foning very simple related problems, by asking themselves 
questions derived from the simplest and most concrete ispects that 
seem to underlie the given situation, 

Di?iioplnf the Sfcille of One f i Miripline 

Finally, students are expected to combine the preceding model and 
processes into the general skills of problem solving as practiced by 
the discipline(s) of choice, 



AronSj At fl, Some thoughts on reasoning capacities Implicitly expected of college students, In J, 
i process instruction * Fhilidilphii s PAi fie Franklin Institute Pries, 1979, 210-213, 



Lochhead k I Clement (Eds,), 



ERIC 



is 



69 



and thinking is complex, but it suffices to say that reasoning processes 
are involved in each. Thorndike (1973) concluded from three seta of 
evidence that, after basic decoding is accomplishedi reading it i rimarily 
an indicator of the general level of an individual T s thinking and 
reasoning processes rather than a separate set of specialised skills* The 
role of reasoning in reading was shown in a slightly different way by 
Baker (1979)* who, after intentionally introducing confusions and 
inconsistent information into reading passages, found the college students 
often failed to recognise logical inconsistencies in reading. Because the 
ability to detect written fallacies in reasoning has been shown to be 
related to reading achievement (Holihan, 1980) , the failure to recognize 
such inconsistencies would undoubtedly hinder eomprehensioni 

Flower and Kayes (1980) have characterised writing as the act of 
balancing a number of simultaneous constraints (e.g,, the demand for 
integrated knowledge, linguistic conventions, and the rhetorical problem 
itself) and suggest that a writer is very much like a busy switchboard 
operator trying to juggle a number of demands on attention and constraints 
on possible actions # (We note in passing that the current analytical 
reasoning items in the GRE General Test appear in many ways to demand the 
capacity to deal with various constraints,) 

Research ir Graduate Education Constituency Surveys , To assess the 
impact of a planned restructuring of the GRE Aptitude, surveys of both 
examinees and institutions were undertaken in 1975, Of the possible new 
measures listed (abstract reasoning, scientific thinking, and study style) 
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abstract reasoniiig was the one most favored by faculty * administrators, 

and students in all fields (Millar & Wild, 1979), 

Examinees were also subsequently asked for their perceptions of 

alternative test item types developed for possible inclusion in a new 

analytical measure, Questions asked of examinees were "Do the questions 

measure abstract reasoning?" "Do the abilities tested seem important?" 

"Are the skills measured different from verbal and quantitative skills?" 

The following results, based on samples of 300 to 1,000 examinees , were 

obtained, As can be seen (Table 3), a majority of examinees thought each 

item type (even those not used in the analytical measure) reflected 

3 

abstract reasoning, Except for the letter sets format, all item types 
were thought to reflect important abilities by a majority of examinees, 
and a majority of examinees rated each item type as measuring something 
other than verbal and quantitative skills* Test takers were least 
convinced that logical reasoning items reflected a different skill. Thus, 
those specifically involved in graduate education appear to recognize the 
importance of abstract reasoning and believe themselves able to 
distinguish among test items with respect to their relevance in measuring 
this attribute. 

Critical Incidents of Graduate Performance , A series of studies by 
Reilly provides another perspective on the skills required of graduate 
students, Reilly (1976) asked graduate faculty in English, chemistry* and 



In any future surveys to determine the face validity of analytical item 
types* it would be useful to include for baseline purposes item types 
that are not intended to measure abstract reasoning, 
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Examinees 1 Reactions to Item Types 



Item Type 


Measure 
abstract 
reasoning? 


seem 
important? 


Different from 
verbal and 
quantitative 
skills? 


Letter sets 


73% 


45% 


80% 


1 2 

Logical reasoning 3 


67% 


66% 


59% 


1 2 

Analytical reasoning ' 


79% 


50% 


80% 


Evaluation of evidence 


76% 


57% 


84% 


Analysis of explanations 


i 1 71% 


57% 


81% 


Logical diagrams" 


83% 


52% 


81% 



Note . From 6% to 10% of students were uncertain whether the item types 
measured abstract reasoning* From 10% to 18% were uncertain whether the 
abilities tested by each item type seemed important. From 4% to 9% were 
uncertain about whether the skills measured were different from verbal and 
quantitative skills* 



^Included in the original analytical test, 
2 

Included in the revised analytical test. 
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psychology to provide examples of specific incidents which, when they 
occurred, caused them either to lower or raise their estimates of the 
competence of graduate students. A total of 32 distinct incidents was 
gathered , a number of which seem related to critical thinking. From these 
incidents, Rei ly (1976) identified the following eight factors: 



1. 


Independence and initiative 


2, 


Conscientiousness 


3, 


Enthusiasm 


4, 


Critical facility 


5, 


Teaching skills 


6, 


Research and experimentation 


7, 


Communication 


8. 


Persistence 



These factors exhibited a relatively high degree of consistency across the 
three fields studied. The critical facility factor was defined by seven 
incidents t 

1* Repeatedly made irrelevant remarks during class or seminar 
discussion* 

2. Talks at great length in class but exhibits l^iitle understanding 
of material on papers and tests* 

3. Was often unable to consider new ideas objectively because of 
strongly held prejudices, 

4. Submitted a paper or report which failed to address the assigned 
issues . 

5. Presented ideas in a seminar, paper s or test in a poorly organised 
and disjointed fashion* 

6* Was unwilling or unable to accept criticism* 
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7* Although able to critlciEe studies with facility was unable to 
suggest better alternatives. 

Although they did not specifically define the critical facility 

factor, several other relevant incidents were also mentioned. These 

include : 

I* Consistently offered weli=founded and constructive criticisms of 
other students 1 presentations* 

2, When making a judgment or rearhing conclusions, this student 
supported his or her position with carefully documented research, 

3, Performed an experiment without making proper checks. 

4* Was unable to formulate a testable hypothesis from a theoretical 



5, When this student asked a question? it was always relevant and 
usually perceptive, 

6* Showed an ability to examine carefully an author T s premises and 
frame of reference before accepting conclusions. 

The last incident (#6) was rated as especially important in all three 

subject fields. In chemistry, it was rated as the eighth most critical 

incident (of 52), and in English and psychology, it was the second most 

important* Of the eight factors listed above , "critical facility 11 was 

ranked as the second most important in English and the fifth most 

important in both chemistry and psychology* Several of these empirically 

determined incidents seem related to the various aspects of critical 

thinking , etc * s that were summarised previously , 
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Training and Measurement of Reasoning 

Training- Frederiksen (1983) reviewed research in cognitive 
psychology on the training of thinking skills, He found some evidence 
that some thinking skills can be taught, but that this possibility is less 
when dealing with ill-structured problems than with well-structured ones. 
He emphasised the importance of further research on ill-structured problem 
solving in order to arrive at better models of the reasoning skills 
involved in problem solving* He also contrasted approaches that attempt 
to teach skills explicitly with those that emphasize the teaching of 
thinking skills by discovery. Finally, he also mentioned the dilemma of 
deciding the level of generality at which skills should be taught* Much 
research in cognitive psychology indicates that learning of problem 
solving skills proceeds more effectively with hands-on problems in 
specific problem domains than with formal instruction in the abstract 
principles of problem solving (Sternberg, 1980)* Snow (1982) and 
Sternberg (1980) have also emphasised the critical role of motivation, 
attitudes, and cognitive style factors in the effective training of 
thinking skills* 

Neimark and Santa (1975) concluded one part of their review on 
thinking and concept attainment by stating that " . . -abstract thinking may 
well be a product of formal education" (p* 192)* And there is a good deal 
of information that formal education does have some impact, Pace (1974), 
as reported by Bowen (1977), reported that a majority of alumni and 
upperclassman, when surveyed, thought that their college experiences had 
benefitted them "very much" or "quite a bit" with respect to their ability 
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to think critically. "Harder" data collected by Keeley, Browne, and 
Kreutzer (1982) and Lehmann and Dressel (1962 • 1963) found changes in 
critical thinking skills from the freshman to the senior years in collage 
in students 1 ability to define problems, recognize stated and unstated 
assumptions, select pertinent informations formulate and select relevant 
hypotheses, and draw valid conclusions. 

More specific, and much briefer, efforts aimed at undergraduate 
students have also reported some success, Feibel (1978) found significant 
changes in reasoning after training based on two different theories of 
cognitive development, and s with only three to six hours of training. Wood 
(I960) had some success in improving the logical reasoning skills (as 
measured by the Watson-Glaser Critical Thinking Appraisal) of 
undergraduate psychology students by using a computer program based on the 
popular game "Mastermind," 

Successful attempts have also bean made to improve the reasoning 
process of medical students in medical problem solving, More experienced 
physicians seem to be more adept than first year medical students in 
making efficient use of information (Mandernach, 1979)* and medical 
students can improve in a relatively short time with appropriate training 
in a diagnostic problem formulation, The critical thinking skills of law 
students have also been improved with relevant intervention, e,g, f with a 
language arts program emphasizing critical thinking and reading (Pachtman, 
1976), That different kinds of training may differentially affect 
critical thinking skills has also been shown — (e fi g s , Agne & Blick, 1972; 
Sorenson, 1966)* The more effective treatments are typically 
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nontraditional ones using laboratory-centered, or experimental, 
data-centered approaches instead of traditional classroom lectures* 
Ennis (1962) suggests that each of three dimensions of critical 
thinking "appear to be learnable somewhat as academic subject matter is 
learnable. , . " (p, 108), these dimensions involve the ability to use 
logical operators, knowledge of criteria for judging statements, and the 
ability to recognize when thet .nough evidence* The other side of the 

coin is that some types of reasoning may be remarkably resistant to 
change, and particular "logical" biases (e,g,, tendency to select 
irrelevant confirmatory instead of possibly relevant disconf irmatory 
evidence) are difficult to eradicate (Mynatt, Doherty, £ Tweney, 1977; 
Wason, & Johnson-Laird, 1972), 

Measurement of Thinking , A 1977 ETS Test Collection bibliography of 
"Measures of Reasoning, Logical Thinking, and Problem Solving Ability" 
contains more than 100 references to measures of these abilities, A brief 
inspection of the entries reveals such subtests as logical reasoning, 
inductive reasoning, conditional reasoning, critical thinking, abstract 
reasoning, analytical reasoning, and syllogistic reasoning. It is clear 
that there has been no shortage of attempts to measure these skills, 

Two of the more interesting measures appropriate for adults are the 
Watson-Glaser Critical Thinking Appraisal (Watson, & Glaser, 1980) and the 
Ennis-Weir Argumentation Test: An Essay Test of Rational Ability (Ennis, 
1982), The Critical Thinking Appraisal includes five subtests of^ 

1. Inference, Discriminating among degrees of truth or falsity of 
inferences drawn from data, 
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2, Recognition of assumptions* Recognizing unstated assumptions or 
presuppositions in given statements or assertions* 

3* Deduction. Determining whether certain conclusions necessarily 
follow from information in given statements or premises* 

4* Interpretation. Weighing evidence and deciding if generalizations 
or conclusions based on the given data are warranted* 

5* Evaluation of arguments. Distinguishing between arguments that 
are strong and relevant and those that are weak or irrelevant* 

This test has probably been researched about as much as any test of 

critical thinking (or related trait) and has been used in as many research 

studies as any other test. Various studies have shewn that the measure 

reflects a dimension that is independent of those tapped by various other 

ability and achievement measures (Follman. Miller* & Hernandez , 1969) and 

by other measures, such as those designed to measure the "structure of 

intellect" (Landis* 1976). Others point out* however* that the various 

measures of critical thinking* including the Critical Thinking Appraisal* 

contain a substantial component of verbal ability (Hartnetts & Willingham* 

1979)* 

The Ennis a -Weir Test is intended to measure the ability to evaluate an 
argument and to formulate in writing an argument in response* The test 
takes the form of a fallacy-laden letter to the editor of a newspaper. 
Examinees read the letter and write an essay evaluating the arguments 
given in each paragraph. A theoretical analysis has been used to validate 
the construct* but to our knowledge no empirical studies have been 
conducted of the construct validity of the measure. 

The point to be made here is that a wide variety of standardized 
measures are available (some □£ which have been researched more 




ERIC 



»63- . 

extensively than others) and that some of these measures may provide 
reference teats against which to gauge the construct validity of the ORE 
analytical measure. 
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Summary 

One objective of this study was to provide a selective review of 
"higher level reasoning" skills* On the face of it, there would seem to 
be little problem in providing auch a review, especially in light of the 
immense amount of educational and psychological research conducted on the 
topic. Difficulties arise, however, in developing a framework for 
defining different kinds of reasoning skills and for showing the 
relationships among different skills, These difficulties have more than 
one origin, but essentially they all center on ambiguities involving 
reasoning as an object of research and assessment* In addition, the 
situation is made more complex when we consider attempts to define and 
investigate the kinds of informal reasoning that people exercise in their 
everyday activities* 

In our judgment, the most important factors affecting the 
interpretation of research on reasoning are the particular description (or 
theory) of mental functioning and the model of reasoning that guide the 
research. This notion is consistent with the views of Messick (1981) and 
Frederiksen (1982) concerning the centrality of both a cognitive theory 
and a performance model in the investigation of construct validity, In 
interpreting research on reasoning we need to appreciate how reasoning is 
defined in terms of cognitive skills and also how task structures in 
criterion settings invoke the use of reasoning skills * Few of the 
previous studies of reasoning offer highly developed descriptions or 
theoretical models of reasoning skills* Without both a cognitive model 
and a performance model of reasoning skills it is difficult to formulate 
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an adequate measurement model that might be used to assess reasoning 
skills, In contrasts although many of the research studies cited do not 
embody a formal description or theory of reasoning, many of the tasks 
studied do show considerable face validity with respect to the reasoning 
demands faced by students in graduate work, especially when compared with 
that of some of the theoretically based work. Often there is a tendency 
for theoretically motivated research to focus on reasoning skills that are 
so tightly constrained in their purpose and application that they exhibit 
little similarity to the kinds of reasoning tasks faced by students in 
graduate school; on the other hand, focus on such "micro" skills might be 
usefully considered, to the extent that it may help predict performance on 
more general reasoning tasks. With these caveats in mind we have 
attempted to review research on reasoning in a way that might contribute 
to studies of the construct validity of the GRE analytical measure, 

One area considered was "critical thinking," which might be deemed as 
the most general of reasoning skills important to successful graduate 
education, Some of the various terms that are used in reference to the 
skills we have in mind range from "reasoning," "judgment, " and the very 
general "thinking 11 (which may be modified by such adjectives as critical, 
abstracts rationals logieals analytical, and creative) s to highly specific 
labels such as "transitive inference," A variety of terms are often 
paired in the general areas for examples thinking and concept attainment 
(Neimark, & Santas 1975), thinking and reasoning (Wason, & Johnson-Lairds 
1970), deductive logic and assumption recognition (Instructional 
Objectives Exchange, 1971) , and critical thinking/critical reading 
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(Pachtman, 1976)* But, critical thinking is also distinguished from a 
variety of other kinds of thinking such as creative thinkings associative 
thinking, and problem solving (Ennis, 1962), 

There is considerable evidence that reasoning skills are viewed as 
extremely important in higher education generally* and in graduate 
education in particulars and further that there are some definite 
expectations with respect to the kinds of reasoning skills that students 
should have. Both of the reviews touch on many of the categories or terms 
used in discussions of reasoning or thinking. 
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IV. An Overview of Skills Measured by the Analytical Section 

Integrating the approaches and findings of previous research in terms 
of their implications for the construct validity of the GRE analytical 
section is not easy, nor can it be done comprehensively. In this section 
we will highlight what appears to be some of the most important cognitive 
and linguistic skills that should be required in solving the two item 
types on the analytical section, Based on our review of the literature * 
Figure 7 summarizes the skills that we believe are the most important for 
solving analytical reasoning and logical reasoning item types. The figure 
is organized so that the reader can readily compare similarities and 
differences in the skills required on the two item types* 

The three classes of skills might be thought of in terms of a 
hierarchy. The top level of this hierarchy concerns examinees 1 general 
knowledge and expertise in multiple choice test taking, This knowledge 
guides the application of general reading comprehension skills represented 
by the second level of the skill hierarchy. At the bottom tier of the 
hierarchy are represented some of the specialized cognitive and linguistic 
skills which are required by an item type. These bottom-level skills are 
utilized in the service of the more general problem solving and reading 
strategies guiding multiple choice test problem solving, 

It is important to note that the skills represented at the top two 
tiers of the hierarchy are important for all aptitude test taking, In the 
GRE General Test these skills are shared in common across not only the two 
analytical section item types, but also across all three sections of the 
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Figure 6 



Skills Hypothetically Required for Solution of Analytical Section Item 



Multiple Choice Test 
Problem Solving and 
Reasoning Strategies 



eral Reading 
Comprehension Skills 



Specialized Cognitive 
and Linguistic Skills 
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Analytical Reasoning Items 



Logical Reasoning Items 



o Understand test instructions 
o Manage time in problem solving 
o Establish the goal of identifying 

only a single correct answer 

option for each item 
o Follow strategies to eliminate 

distractor options when necessary 



o Vocabulary recognition 
o Sentence recognition 
o Paragraph structure reeog 
(limited) 
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o Recognizing algebraic-like rela- i 
tienships and properties of these 
relationships 

- Conservation of quantity 

- Reflexlvity 

- Symmetry 

- Transitivity i 

- Class inclusion/non-inclusion 
and set membership/non^membershlp 

o Ability to reason deductively/ 
inductively about relationships, 
including ability to enact serial 
operations 

o Ability to keep track of problem 
information and alternative models 
of a problem (short-term memory span) 

o Ability to use diagrams to encode 
problem information (optional) 

o Ability to generate combinations and 
to search problem space for counter* 
examples 



o Understand test instructions 
o Manage time in problem solving 
o Establish the goal of identifying 

only a single correct answer 

option for each item 
o Follow strategies to eliminate 

distractor options when necessary 

o Vocabulary recognition 
o Sentence recognition 
o Paragraph structure recognition 
(important) 

o Recognizing the structure of an 
argument 

■ Isolate premises 

- Isolate conclusions 

- Isolate noncritical information 
in an argument 

o Evaluating the validity of conclusions 

- Understanding how logical entail- 
ment is qualified in an argument 

- Reasoning deductively 

■ Reasoning inductively 
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General Test, There is probably an important amount of overlap in the com- 
mon, general skills required to work different sections and item types. This 
overlap contributes to the intercorrelation of performance across all item 
types. In addition, systematic overlap may occur across sections and item 
types because of commonality in specific cognitive and linguistic skills, 
This matter will be discussed in the ensuing overview of the specific cog- 
nitive and linguistic demands of each analytical section item type, Before 
starting this discussion it is useful to remember that at present we lack 
empirical evidence verifying the contribution of the cognitive and linguistic 
skills mentioned to performance on each analytical item type, A goal of the 
discussion is thus to identify skills that could become part of an empirical 
research study on the construct validity of analytical section item types. 

The analytical reasoning item type and quantitative General Test items 
share a common emphasis on reasoning about relationships among variables, 
While analytical reasoning items do not require manipulation of numbers for 
their solution, the relationships among variables in these problems emphasize 
algebraic properties important to quantitative reasoning* For example, 
relationships of transitivity/intransitivity, synmie try /a symmetry, and 
conservation of quantity are often critical to solving analytical reasoning 
items As with quantitative items, the solution of analytical reasoning 
items requires noticing key relationships among variables and interpretation 
of questions in terms of these relationships, The difficulty of problems has 
been found to increase as the number of relationships required to work items 
increases. 
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There ar^ a number df 



Information processing skiil@ which teeeni critical 



to working analytical r#a .soning items* Ability to re^gantfith serial oper- 
ations seems important 3 a s does short-term memory capacity* afl^S the degree of 
proficiency in these skil Is would seem to become BiQre fimpoftafltz as problems 
become more complex and d ±f f icult . Ability to represent lnfofftmation spa- 
tially in terms of diagr^a^^is may be important^ and this iaport^teace is likely 
to be a function of the d : ±f f iculty of problems and individual ^examinee f s 
preference or need for dl^- -agramming strategies* Representation with other 
symbol systems is an alee - -rnative strategy for soma eaca^pineett . 

The test development ^ apeeif ications for the logical reasont=Lng item type 
are explicit as to the r*e^asoning skills which are targeted for assessment by 
each item. The reasoning abilities tested emphasize skills in analyzing 
verbal argumenti and in reevaluating the degree to vhich argument's are 
supported by statements, Vocabulary and sentence recognition &^kills seem 
critical to the item type ^ 3 since examinees must (a) dative a pr^ecise under- 
standing of the propositlsons represented by sentences and (b) uimnder stand the 
way in which vocabulary q^eialifies the interpretation off atategieL^nts * 
Examinees must be able to discern how the information tJ a Pa#& «age is 
organized and connected toogether to form a logical arliunt* T^huSj it is 
necessary for examinees too understand both the discourse struct ■— ure of item 
passages and the semantic and logical relationships affwif ifthce, ^nces* They 
must also be aWa to disti^Lnguish information that is p^rt of # set of 
premises from information that is implicitly or explicitly all&^gad to be a 
conclusion based on preml^Mes » Examinees' ability to detect g^a_T_Tnmatical 
structures and vocabulary terms signifying logical ent^Jlpnt w^ould seem to 
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be of critical importance. Examinees also have to distinguish inf o^^Tnation 
that is not relevant to conclusions that might bs drawn from a passage, That 
these items correlate more strongly with verbal ability than do analytical 
reasoning items is thus not surprising. They represent a crucial labset of 
reading comprehension skills that may warrant separate testing, if mot 
separate reporting* 
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V. Summary 

The objective of this paper was to work towards a batter understanding of 
the current version of the GRE analytical measure as an indicator of cogni- 
tive skills that are important for success in graduate education* This 
better understanding was to be accomplished within the context of the con= 
siderable developmental research that preceded the selection of test item 
types for the original version of the GRE analytical measures which was 
introduced in 1977. 

However, the choice and mix of item types for the original measure was 
dictated largely by traditional psychometric considerations, as was the 
measure's construct validity. Since the analytical measure was Introduced, 
Messieks Frederiksen, and others have advanced the conceptions of construct 
validity, stressing the role of theory in understanding both the subskills 
that are critical to performance on a test and also the relationships of test 
performance to performance or "real world* 1 criteria. Recent advances in 
cognitive psychology have also contributed to educational and psychological 
testing by focusing on the specific processes that underlie the solution of 
test items* 

This paper has described the current version of the GRE analytical meas- 
ure, its developmental history, and the research that led to revisions of the 
original measure. Selective reviews of literature have been presented for 
two relevant areas* One review focused on the psychological and cognitive 
science research on reasoning and problem solving. The other involved the 
somewhat more general educational research on the role of thinking or 
reasoning in higher education, 
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The results of these reviews suggested a number of things. First, there 
is a substantial amount of literature in these areas. However, there do not 
seem to be any comprehensive educational or psychological theories for rea- 
soning, thinking, or problem solving. The absence of any well=developed 
theories renders the measurement of reasoning skills significantly more 
difficult. On the other hand, these reviews strongly suggest, at least indi- 
rectly, the importance of higher^level reasoning skills in higher education, 
This paper also has analysed each of the two item types that are used in 
the current version of the analytical measure, This analysis provides an 
overview of the major reasoning skills that seem to be required for the 
successful solution of these item types* 

A major conclusion of this paper is that the current version of the GRE 
analytical measure appears to reflect skills that are important for success 
in graduate education, but that these skills constitute only a subset of many 
other important analytical skills. Further empirical research is recommended 
in order to document more precisely the demands of graduate education on 
students' analytical abilities. 
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Let us consider a simulated atmlysi^s of one w-=ay in which the 
analytical reasoning situation in Pigur— e 1 light Ke interpreted and used 
to answer questions. This analysis cfta^racterizes some of the cognitive 
operations required for the solution of ^ quistiona — 

I- With 7 individuals, there are - 128 diff errant possible 

combinations to consider? so - any attempsc to list or diagram all 
possible combinations would W m inpraeti<— ^al . An initial planning 
or search control decision is vital if «ny efficient strategy is 
to be adopted. Thus, ruling *out impogeMlbilities rather than 
systematically constructing alll poiisibiM=±ties must be seen as a 
^aiid solution strategy* A ecsogiiitive sfcyle that requires more 
closure than the probleGi golti^^leii demanc^tts eoulrl be a great 
disadvantage at this initial sstafe* 
11* Correct encoding of the itifora^nation does» not require knowledge 
of technical terms* but does r^equire pre&cise understanding of 
the logical meaning of ordinac^ry terms of" relationship* Such 
errors as interpreting ,f Ftan ^goei to a r— restaurant only on 
Wednesdays," to mean "Fran al&yays goes o ti Wednesdays, 11 or "Helen 
will not go to a restaurant unless Fran jIso goes," to mean, "If 
Fran goes, then Helen must go, _ fI are clea — arly very serious, but 
research suggests that ev^n #dS^aneed und^^ergraduates are prone to 
waking such errors with surprising frequ^ency* 
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III* In analytical reasoning problems In which a complex structure 
must be modeled, finding a representation and convenient 
notation can be important. In this problem, only a few 
exclusion rules are needed, and it is not necessary to depart 
from verbal representation, if the rules have been correctly 
encoded* It may be helpful for some to abbreviate this verbal 
representation, as in! 

1. No F if not Wed. 

2. No G if K 
No K if G 

3* No H if No F 

4. No J and M if No G (But J or M ok if No G) 
5* No K if Rest* 0 
IV* By applying transitive inference, several additional rules may 
be derived from these, e*g*; 

No H if not Wed, (16 3) 

No J and M if K (2 & 4) 
V- Specializing to question It If six individuals have lunch in 
one of the restaurants on a Wednesday, we see that it is not 
necessary to work out all the exclusions to answer the question* 
We may merely trace through a branch until it results in fewer 
than six workers being available and then go to the next 
possibility. Here a kind of test^wiseness involving the 
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expectation that one and only one answer choice is correct is 
important. If a particular chain excludes only one individuals 
and that individual is among the response choices, the search 
may be terminated without checking all other possibilities* It 
may be prudent to check other possibilities as a check on one f s 
reasoning, but if another solution is found, it is evidence of a 
mistake in representation or inference (or conceivably, of an 
error in the question) rather than of two intended answer keys. 
Thus we may work through the conditions until we find an Individual 
whose presence implies the absence of at least two others, and we will 
have the solution. In this case, we are given that it Is Wednesday, so by 
1) Fran is not excluded. 

By 2), either Gladys or Karl is excluded, so Fran must be included if 
there are to be six Individuals at lunch. Assume Gladys is excluded, 

By 3), Helen is not excluded, since Fran is included, indeed, Helen 
must be Included to make the total six. 

By 4), Juan or Mark is also excluded if Gladys is excluded. 
Therefore, the assumption that Gladys is excluded makes it impossible to 
include six individuals, and Gladys must be included. Therefore, by 2) , 
Karl must be excluded, and we have a solution. Checking this against 5) 
we see that excluding Karl does not lead to any inconsistency, so we 
select response A, This argument has employed an indirect argument, 
assuming one alternative and showing that it leads to an impossible 
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outcoflii go that: the ot^ier alternative must be the case. This form of 
reasoning ts extremely important in analytical reasoning items , and does 
not appear to be well-studied in the research literature, other than in 
Wason and Jdinson-Laird w m documentation of subjects 1 tendency to seek 
confirftiing* rather than di scon firming evidence* A search control strategy 
that recognizes that th^s search is over once it has been shown that one of 
two e^hau#t ive alternat^Lves cannot be the case is more efficient than one 
that continues to check possibilities after the problem i^ solved, A 
construct validation of this item type should explicitly examine the 
import%n£g of this riasoning paradigm in the tasks confronting graduate 
students* 

The aeCcDtl problem ^^xi this sample set asks for the greatest number of 
individuals who can go out to lunch together on a Tuesday at one of the 
three restaurants, Altfciough it deals with maximizing the number of 
individuals present , rafcher than with finding conditions for exactly one 
individual to be absent^ the solution is very strongly related to problem 
i* 

By 1) t Fran is eKclimded, and so, by 3) Helen is also, so the number is 
less than six. 

By the saisi reasonir»ig used in the first problem, excluding Gladys also 
excludes Jueiior Mark, t^xit excluding Karl excludes no one else, To 
maximize the number, we ^gain exclude Karl, so there can be no more than 
4* Th^re ate no other conditions, so the greatest number is 4. 
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Problem 3 involves additional reasoning steps* If Juan iend Mark eat 
lunch with a group of co-workers, we know by 4) that Gladys itaust be there, 
and again by 2) that poor Karl is shut out again* Therefore, _ Statement I, 
"The group can include no more than three men* 11 must ba t r u^. Unless it 
is Wednesday, neither Fran nor Helen can be included, On the - other hands 
if it is Wednesday, Fran may or may not attend, and if Fran a?ittends, Helen 
may or may not join her. Therefore, depending on circumstflnc*_es which may 
vary within the situation described* there may be one, two, ot-r three women 
present, and Statement II, "The group can include only two votraien," is 
false, because we have found a counterexample in a permit sibl^-e case with 
all three able to be there. Since we already know that Karl tie excluded, 
his adversion to restaurant 0 is not relevant, and Statement till, "The 
group cannot eat at restaurant 0," is false, Note the need icrao deal with 
multiple negation in this solution, Karl will not go to restaurant 0, 
Karl is not in the group. Therefore, it is not true that the group cannot 
eat at restaurant 0, Dealing with negation is particularly tr^roublesome 
for many students, and this question gets at this important reasoning 
skill without resorting to awkward wording, All three quest i&ons require 
the examinee to have a clear understanding of the difference iiemong 
statements that must necessarily be true, statements that be true, and 

statements that cannot be true. In questions 2 and 3 certain statements 
that may be true must be used to demonstrate the falsity of ge-^neraliza- 
tions. However, in no case is it necessary to generate and ga-earch the 
entire list of combinations that may be true* The ability to % go to just 
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those relevant cases that could disconf irm a proposition reflects an un- 
derstanding of how deductive logic proceeds. Additional confirming 
instances ,do not prove a statement* One must show that there are no 
disconf inning cases* Apparently analytical reasoning items tap this 
understanding without resort to mathematical or logical terminology or 
symbolism. This ability to "'get to the heart of the matter 11 may be more 
related to induction or even to an intuition developed by practice in 
problem solving than are such more commonly emphasized components such as 
transitive inference or rules for negation* It can be most effectively 
tested with problems such as this example In which there are numerous 
possibilities not completely determined by the conditions. Because they 
are under— determined (but not ill—structured) s such problems usually are 
not easily diagrammed. It may be that some of the disagreements which 
recur about the value of diagrams in solving logic problems could be 
resolved by clearer specification of the degree to which the problems are 
under— determined in this sense, Certainly any construct validity study 
for analytical reasoning skills should make this variable explicit, 

A much more detailed account of the steps involved in solving these 
sample problems is of course well within the reach of s and indeed 
preferred by cognitive psychologists, As one focusses on more microscopic 
procedural levels such as encoding and representing each phrase in the 
stimulus* the links to research on thinking in educational contexts become 
longer and thinner. Although more detailed modelling might have 
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considerable value for test development , it is not deemed appropriate for 
construct validation, in which links to more global variables emphasized 
in previous research and thinking about thinking are the major foci* On 
the other hand, very global accounts, based on taxonomical cognitive 
levels such as "comprehension, application, or evaluation," or on Baconian 
models of scientific method, with similar categories (eg*, observe, 
diagnose, plan, implement, evaluate) are so general as not even to admit 
of agreement among judges as to the categorization of items * Such very 
global specifications have failed to stand up to empirical study of the 
actual structure of problem-solving activity* If: is hoped that the level 
of description attempted here is near that target* 
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